<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><br><div><div>On 4 Feb 2010, at 14:42, Philippe Van Dyck wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><br><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"><div style="word-wrap:break-word"><div><div class="im"><div><br></div></div><div>I'm trying to think of how this can be. Worker threads adding data, adding stuff to the async cache store queue for flushing. The eviction thread removing stuff from the data container *only*. </div>
<div><br></div><div>*Perhaps* what you see is a race where you have:</div><div><br></div><div>1 add item to data container</div><div>2 enqueue in async cache store for storage</div><div>3 evict in memory</div><div>4 attempt a get</div>
</div></div></blockquote><div><br></div><div>Actually, the more I think about it, the transaction probably fails because the datacontainer has been emptied (get(key) does not work anymore).... But it is definitely not supposed to die silently !</div>
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"><div style="word-wrap:break-word"><div><div><br></div><div>where steps 1 - 4 happen *before* the async cache store can flush its queue to disk. So this would result in the thread in 4 consulting the data container, not finding the entry, then checking the cache store and not finding it there either since it hasn't been flushed yet. </div>
<div><br></div><div>Now IMO this is normal behaviour - the price you pay for asynchronously writing to a store. But perhaps this window can be </div></div></div></blockquote><div><br></div><div>Am I missing something ? Loosing data is something I cannot afford ! I Plan to use this store as a *permanent* one... I have no backup ! (Actually S3 is the backup) - So, no, I don't want this ... at any price ;-)</div></div></blockquote><div><br></div><div>Then set <async enabled="false" /> in your cache store config. :-)</div><br><blockquote type="cite"><div class="gmail_quote">
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"><div style="word-wrap:break-word"><div><div>reduced by looking through the async queue as well, before checking the underlying store. But as I said, this just reduces the size of this window and not eliminate it altogether, since this is async and there is no guarantee that the cache store has finished writing internally (e.g., an fsync() operation or in the case of S3, Amazon's eventual consistency model).</div>
<div><div></div><div class="h5"><br><div><br></div></div></div><div>Why should eviction be transactional? I don't need eviction to be an all-or-nothing, reversible event. :) If an entry gets evicted, cool. If not (for whatever reason), too bad, move on to the next evictable entry. </div>
</div></div></blockquote><div><br></div><div>You are right, we don't want to rollback evictions... but maybe we should use a priority queue to be sure that evictions are done after any other command ? Doesn't it solve it all ? </div>
<div><br></div><div>1) The eviction thread runs (we could lower the priority of this thread too)</div><div>2) It fills a queue of keys to evict</div><div>3) The async queue is prioritized and evicts entries ... when there is nothing else to do (suddenly it looks like garbage collecting)</div></div></blockquote><div><br></div><div>That is a possibility. But I don't expect to be making any drastic changes to the existing eviction code anymore. Don't know if you have been following discussions re: LIRS, lock amortization, etc., but Vladimir is working on some very interesting self-evicting, bounded data containers which would mean that the eviction threads, etc all get ripped out.</div><br><blockquote type="cite"><div class="gmail_quote">
<div> </div><div><br></div><div>WDYT ?</div><div><br></div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"><div style="word-wrap:break-word"><div><div>
<br></div><div>Cheers</div><div>Manik</div><div><div></div><div class="h5"><br><blockquote type="cite"><div>
<div><br></div><div><div>Looks like a design issue ? WDYT ?</div><div><br></div></div><div><br></div><div>Cheers,</div><div><br></div><div>Phil</div><div><br></div><div><br><div class="gmail_quote">On Thu, Feb 4, 2010 at 10:44 AM, Manik Surtani <span dir="ltr"><<a href="mailto:manik@jboss.org" target="_blank">manik@jboss.org</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word">That is strange since there is no correlation between eviction and the synchronicity of cache stores. Have you got a reproducible test for this?<div>
<br></div><div>Cheers</div><div>Manik</div><div><div></div><div><div><div><br><div><div>On 3 Feb 2010, at 18:37, Philippe Van Dyck wrote:</div><br><blockquote type="cite">Thanks Manik,<div><br></div><div>I have a another problem with eviction, it seems to destroy cache entries, <i>only when I use async</i>.</div>
<div><br></div><div>Of course, all updates are transactional.</div><div><br></div>
<div>Where should I search for clues ? Any idea ?</div><div><br></div><div>Here is my config:</div><div><br></div><div><div><?xml version="1.0" encoding="UTF-8"?></div><div><br></div><div><infinispan xmlns:xsi="<a href="http://www.w3.org/2001/XMLSchema-instance" target="_blank">http://www.w3.org/2001/XMLSchema-instance</a>"</div>
<div><span style="white-space:pre"> </span>xmlns="urn:infinispan:config:4.0"></div><div><span style="white-space:pre"> </span><global></div><div><span style="white-space:pre"> </span><transport</div>
<div><span style="white-space:pre"> </span>transportClass="org.infinispan.remoting.transport.jgroups.JGroupsTransport"></div><div><span style="white-space:pre"> </span><properties></div>
<div><span style="white-space:pre"> </span><property name="configurationFile" value="jgroups.xml" /></div><div><span style="white-space:pre"> </span></properties></div>
<div><span style="white-space:pre"> </span></transport></div><div><br></div><div><span style="white-space:pre"> </span></global></div><div><br></div><div><span style="white-space:pre"> </span><namedCache name="qi4j"></div>
<div><span style="white-space:pre"> </span><transaction</div><div><span style="white-space:pre"> </span>transactionManagerLookupClass="org.infinispan.transaction.lookup.DummyTransactionManagerLookup" /></div>
<div><span style="white-space:pre"> </span><clustering mode="distribution"></div><div><span style="white-space:pre"> </span><l1 enabled="true" lifespan="100000" /></div>
<div><span style="white-space:pre"> </span><hash numOwners="1" rehashRpcTimeout="120000" /></div><div><span style="white-space:pre"> </span></clustering></div>
<div><br></div><div><span style="white-space:pre"> </span><loaders passivation="false" shared="true" preload="false"></div><div><br></div><div><span style="white-space:pre"> </span><loader class="org.infinispan.loaders.file.FileCacheStore"</div>
<div><span style="white-space:pre"> </span>fetchPersistentState="false" ignoreModifications="false"</div><div><span style="white-space:pre"> </span>purgeOnStartup="true"></div>
<div><span style="white-space:pre"> </span><properties></div><div><span style="white-space:pre"> </span><property name="location" value="/tmp" /></div>
<div><span style="white-space:pre"> </span></properties></div><div><span style="white-space:pre"> </span><async enabled="true" threadPoolSize="3" /></div>
<div><span style="white-space:pre"> </span></loader></div><div><br></div><div><span style="white-space:pre"> </span></loaders></div><div><span style="white-space:pre"> </span></div>
<div><span style="white-space:pre"> </span><deadlockDetection enabled="true" spinDuration="1000"></deadlockDetection></div><div><br></div><div><span style="white-space:pre"> </span><eviction strategy="FIFO" wakeUpInterval="1000" maxEntries="10" /></div>
<div><br></div><div><span style="white-space:pre"> </span><unsafe unreliableReturnValues="true" /></div><div><br></div><div><span style="white-space:pre"> </span></namedCache></div>
<div></infinispan></div><div><br></div></div><div><br></div><div>phil</div><div><br></div><div><br><br><div class="gmail_quote">On Wed, Feb 3, 2010 at 6:42 PM, Manik Surtani <span dir="ltr"><<a href="mailto:manik@jboss.org" target="_blank">manik@jboss.org</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word">Ugh, good point. I thought the unit tests would have trapped a dumb-ass mistake like this.<div>
<br></div><div>The reason for transforming the name of the bucket is that we usually use hashcodes as the bucket name, which can take Integer.MIN_VALUE to Integer.MAX_VALUE. These are then translated into Strings, and this becomes the name of the storage unit, e.g., 12345.bucket in the FileCacheStore. Now filesystems are happy to accept a -12345.bucket but certain cloud storage providers barf when encountering the '-' character. Hence the transformation to A12345.bucket in some cases.</div>
<div><br></div><div>Cheers</div><div>Manik</div><div><br></div><div>PS: pushing up a new snapshot as I type, containing this fix + lower verbosity on eviction-related lock timeouts.</div><div><br><div><div><div></div><div>
<div>On 3 Feb 2010, at 17:16, Philippe Van Dyck wrote:</div><br></div></div><blockquote type="cite"><div><div></div><div>And BTW, why do it ?<div><br></div><div>p<br><br><div class="gmail_quote">---------- Forwarded message ----------<br>
From: <b class="gmail_sendername">Philippe Van Dyck</b> <span dir="ltr"><<a href="mailto:pvdyck@gmail.com" target="_blank">pvdyck@gmail.com</a>></span><br>
Date: Wed, Feb 3, 2010 at 6:15 PM<br>Subject: CloudCacheStore Bug<br>To: infinispan -Dev List <<a href="mailto:infinispan-dev@lists.jboss.org" target="_blank">infinispan-dev@lists.jboss.org</a>><br><br><br>Hi all,<div>
<br></div><div>
there is a bug in CloudCacheStore that makes me feel like I am the only one using it ;-)</div><div><br></div><div>in CR4 : if you change the "-" sign to "A" in getBucketName ... you need to do the opposite somewhere (or call it every time) ;-)</div>
<div><br></div><div>WDYT ?</div><div><br></div><font color="#888888"><div>p</div>
</font></div><br></div></div></div>
_______________________________________________<br>infinispan-dev mailing list<div><br><a href="mailto:infinispan-dev@lists.jboss.org" target="_blank">infinispan-dev@lists.jboss.org</a><br></div><a href="https://lists.jboss.org/mailman/listinfo/infinispan-dev" target="_blank">https://lists.jboss.org/mailman/listinfo/infinispan-dev</a></blockquote>
</div><br><font color="#888888"><div>
<span style="border-collapse:separate;color:rgb(0, 0, 0);font-family:Helvetica;font-size:12px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:auto;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><span style="border-collapse:separate;color:rgb(0, 0, 0);font-family:Helvetica;font-size:12px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><div style="word-wrap:break-word">
<div>--</div><div>Manik Surtani</div><div><a href="mailto:manik@jboss.org" target="_blank">manik@jboss.org</a></div><div>Lead, Infinispan</div><div>Lead, JBoss Cache</div><div><a href="http://www.infinispan.org/" target="_blank">http://www.infinispan.org</a></div>
<div><a href="http://www.jbosscache.org/" target="_blank">http://www.jbosscache.org</a></div><div><br></div></div></span><br></span><br>
</div>
<br></font></div></div><br>_______________________________________________<br>
infinispan-dev mailing list<br>
<a href="mailto:infinispan-dev@lists.jboss.org" target="_blank">infinispan-dev@lists.jboss.org</a><br>
<a href="https://lists.jboss.org/mailman/listinfo/infinispan-dev" target="_blank">https://lists.jboss.org/mailman/listinfo/infinispan-dev</a><br></blockquote></div><br></div>
_______________________________________________<br>infinispan-dev mailing list<br><a href="mailto:infinispan-dev@lists.jboss.org" target="_blank">infinispan-dev@lists.jboss.org</a><br><a href="https://lists.jboss.org/mailman/listinfo/infinispan-dev" target="_blank">https://lists.jboss.org/mailman/listinfo/infinispan-dev</a></blockquote>
</div><br><div>
<span style="border-collapse:separate;color:rgb(0, 0, 0);font-family:Helvetica;font-size:12px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:auto;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><span style="border-collapse:separate;color:rgb(0, 0, 0);font-family:Helvetica;font-size:12px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><div style="word-wrap:break-word">
<div>--</div><div>Manik Surtani</div><div><a href="mailto:manik@jboss.org" target="_blank">manik@jboss.org</a></div><div>Lead, Infinispan</div><div>Lead, JBoss Cache</div><div><a href="http://www.infinispan.org/" target="_blank">http://www.infinispan.org</a></div>
<div><a href="http://www.jbosscache.org/" target="_blank">http://www.jbosscache.org</a></div><div><br></div></div></span><br></span><br>
</div>
<br></div></div></div></div></div><br>_______________________________________________<br>
infinispan-dev mailing list<br>
<a href="mailto:infinispan-dev@lists.jboss.org" target="_blank">infinispan-dev@lists.jboss.org</a><br>
<a href="https://lists.jboss.org/mailman/listinfo/infinispan-dev" target="_blank">https://lists.jboss.org/mailman/listinfo/infinispan-dev</a><br></blockquote></div><br></div></div>
_______________________________________________<br>infinispan-dev mailing list<br><a href="mailto:infinispan-dev@lists.jboss.org" target="_blank">infinispan-dev@lists.jboss.org</a><br><a href="https://lists.jboss.org/mailman/listinfo/infinispan-dev" target="_blank">https://lists.jboss.org/mailman/listinfo/infinispan-dev</a></blockquote>
</div></div></div><div><div></div><div class="h5"><br><div>
<span style="border-collapse:separate;color:rgb(0, 0, 0);font-family:Helvetica;font-size:12px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:auto;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><span style="border-collapse:separate;color:rgb(0, 0, 0);font-family:Helvetica;font-size:12px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><div style="word-wrap:break-word">
<div>--</div><div>Manik Surtani</div><div><a href="mailto:manik@jboss.org" target="_blank">manik@jboss.org</a></div><div>Lead, Infinispan</div><div>Lead, JBoss Cache</div><div><a href="http://www.infinispan.org/" target="_blank">http://www.infinispan.org</a></div>
<div><a href="http://www.jbosscache.org/" target="_blank">http://www.jbosscache.org</a></div><div><br></div></div></span><br></span><br>
</div>
<br></div></div></div><br>_______________________________________________<br>
infinispan-dev mailing list<br>
<a href="mailto:infinispan-dev@lists.jboss.org">infinispan-dev@lists.jboss.org</a><br>
<a href="https://lists.jboss.org/mailman/listinfo/infinispan-dev" target="_blank">https://lists.jboss.org/mailman/listinfo/infinispan-dev</a><br></blockquote></div><br>
_______________________________________________<br>infinispan-dev mailing list<br><a href="mailto:infinispan-dev@lists.jboss.org">infinispan-dev@lists.jboss.org</a><br>https://lists.jboss.org/mailman/listinfo/infinispan-dev</blockquote></div><br><div>
<span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div>--</div><div>Manik Surtani</div><div><a href="mailto:manik@jboss.org">manik@jboss.org</a></div><div>Lead, Infinispan</div><div>Lead, JBoss Cache</div><div><a href="http://www.infinispan.org">http://www.infinispan.org</a></div><div><a href="http://www.jbosscache.org">http://www.jbosscache.org</a></div><div><br></div></div></span><br class="Apple-interchange-newline"></span><br class="Apple-interchange-newline">
</div>
<br></body></html>