Ugh, good point. I thought the unit tests would have trapped a dumb-ass mistake like this. The reason for transforming the name of the bucket is that we usually use hashcodes as the bucket name, which can take Integer.MIN_VALUE to Integer.MAX_VALUE. These are then translated into Strings, and this becomes the name of the storage unit, e.g., 12345.bucket in the FileCacheStore. Now filesystems are happy to accept a -12345.bucket but certain cloud storage providers barf when encountering the '-' character. Hence the transformation to A12345.bucket in some cases. Cheers Manik PS: pushing up a new snapshot as I type, containing this fix + lower verbosity on eviction-related lock timeouts. On 3 Feb 2010, at 17:16, Philippe Van Dyck wrote: And BTW, why do it ? p ---------- Forwarded message ---------- From: Philippe Van Dyck <pvdyck(a)gmail.com&gt; Date: Wed, Feb 3, 2010 at 6:15 PM Subject: CloudCacheStore Bug To: infinispan -Dev List <infinispan-dev(a)lists.jboss.org&gt; Hi all, there is a bug in CloudCacheStore that makes me feel like I am the only one using it ;-) in CR4 : if you change the "-" sign to "A" in getBucketName ... you need to do the opposite somewhere (or call it every time) ;-) WDYT ? p _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev -- Manik Surtani manik(a)jboss.org Lead, Infinispan Lead, JBoss Cache http://www.infinispan.org http://www.jbosscache.org _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

That is strange since there is no correlation between eviction and the synchronicity of cache stores. Have you got a reproducible test for this? Cheers Manik On 3 Feb 2010, at 18:37, Philippe Van Dyck wrote: Thanks Manik, I have a another problem with eviction, it seems to destroy cache entries, *only when I use async*. Of course, all updates are transactional. Where should I search for clues ? Any idea ? Here is my config: <?xml version="1.0" encoding="UTF-8"?> <infinispan xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="urn:infinispan:config:4.0"> <global> <transport transportClass="org.infinispan.remoting.transport.jgroups.JGroupsTransport"> <properties> <property name="configurationFile" value="jgroups.xml" /> </properties> </transport> </global> <namedCache name="qi4j"> <transaction transactionManagerLookupClass="org.infinispan.transaction.lookup.DummyTransactionManagerLookup" /> <clustering mode="distribution"> <l1 enabled="true" lifespan="100000" /> <hash numOwners="1" rehashRpcTimeout="120000" /> </clustering> <loaders passivation="false" shared="true" preload="false"> <loader class="org.infinispan.loaders.file.FileCacheStore" fetchPersistentState="false" ignoreModifications="false" purgeOnStartup="true"> <properties> <property name="location" value="/tmp" /> </properties> <async enabled="true" threadPoolSize="3" /> </loader> </loaders> <deadlockDetection enabled="true" spinDuration="1000"></deadlockDetection> <eviction strategy="FIFO" wakeUpInterval="1000" maxEntries="10" /> <unsafe unreliableReturnValues="true" /> </namedCache> </infinispan> phil On Wed, Feb 3, 2010 at 6:42 PM, Manik Surtani <manik(a)jboss.org&gt; wrote: > Ugh, good point. I thought the unit tests would have trapped a dumb-ass > mistake like this. > > The reason for transforming the name of the bucket is that we usually use > hashcodes as the bucket name, which can take Integer.MIN_VALUE to > Integer.MAX_VALUE. These are then translated into Strings, and this becomes > the name of the storage unit, e.g., 12345.bucket in the FileCacheStore. Now > filesystems are happy to accept a -12345.bucket but certain cloud storage > providers barf when encountering the '-' character. Hence the > transformation to A12345.bucket in some cases. > > Cheers > Manik > > PS: pushing up a new snapshot as I type, containing this fix + lower > verbosity on eviction-related lock timeouts. > > On 3 Feb 2010, at 17:16, Philippe Van Dyck wrote: > > And BTW, why do it ? > > p > > ---------- Forwarded message ---------- > From: Philippe Van Dyck <pvdyck(a)gmail.com&gt; > Date: Wed, Feb 3, 2010 at 6:15 PM > Subject: CloudCacheStore Bug > To: infinispan -Dev List <infinispan-dev(a)lists.jboss.org&gt; > > > Hi all, > > there is a bug in CloudCacheStore that makes me feel like I am the only > one using it ;-) > > in CR4 : if you change the "-" sign to "A" in getBucketName ... you need > to do the opposite somewhere (or call it every time) ;-) > > WDYT ? > > p > > _______________________________________________ > infinispan-dev mailing list > > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev > > > -- > Manik Surtani > manik(a)jboss.org > Lead, Infinispan > Lead, JBoss Cache > http://www.infinispan.org > http://www.jbosscache.org > > > > > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev > _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev -- Manik Surtani manik(a)jboss.org Lead, Infinispan Lead, JBoss Cache http://www.infinispan.org http://www.jbosscache.org _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

I'm trying to think of how this can be. Worker threads adding data, adding stuff to the async cache store queue for flushing. The eviction thread removing stuff from the data container *only*. *Perhaps* what you see is a race where you have: 1 add item to data container 2 enqueue in async cache store for storage 3 evict in memory 4 attempt a get

where steps 1 - 4 happen *before* the async cache store can flush its queue to disk. So this would result in the thread in 4 consulting the data container, not finding the entry, then checking the cache store and not finding it there either since it hasn't been flushed yet. Now IMO this is normal behaviour - the price you pay for asynchronously writing to a store. But perhaps this window can be

reduced by looking through the async queue as well, before checking the underlying store. But as I said, this just reduces the size of this window and not eliminate it altogether, since this is async and there is no guarantee that the cache store has finished writing internally (e.g., an fsync() operation or in the case of S3, Amazon's eventual consistency model). Why should eviction be transactional? I don't need eviction to be an all-or-nothing, reversible event. :) If an entry gets evicted, cool. If not (for whatever reason), too bad, move on to the next evictable entry.

Cheers Manik Looks like a design issue ? WDYT ? Cheers, Phil On Thu, Feb 4, 2010 at 10:44 AM, Manik Surtani <manik(a)jboss.org&gt; wrote: > That is strange since there is no correlation between eviction and the > synchronicity of cache stores. Have you got a reproducible test for this? > > Cheers > Manik > > On 3 Feb 2010, at 18:37, Philippe Van Dyck wrote: > > Thanks Manik, > > I have a another problem with eviction, it seems to destroy cache entries, > *only when I use async*. > > Of course, all updates are transactional. > > Where should I search for clues ? Any idea ? > > Here is my config: > > <?xml version="1.0" encoding="UTF-8"?> > > <infinispan xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" > xmlns="urn:infinispan:config:4.0"> > <global> > <transport > > transportClass="org.infinispan.remoting.transport.jgroups.JGroupsTransport"> > <properties> > <property name="configurationFile" value="jgroups.xml" /> > </properties> > </transport> > > </global> > > <namedCache name="qi4j"> > <transaction > transactionManagerLookupClass="org.infinispan.transaction.lookup.DummyTransactionManagerLookup" > /> > <clustering mode="distribution"> > <l1 enabled="true" lifespan="100000" /> > <hash numOwners="1" rehashRpcTimeout="120000" /> > </clustering> > > <loaders passivation="false" shared="true" preload="false"> > > <loader class="org.infinispan.loaders.file.FileCacheStore" > fetchPersistentState="false" ignoreModifications="false" > purgeOnStartup="true"> > <properties> > <property name="location" value="/tmp" /> > </properties> > <async enabled="true" threadPoolSize="3" /> > </loader> > > </loaders> > <deadlockDetection enabled="true" > spinDuration="1000"></deadlockDetection> > > <eviction strategy="FIFO" wakeUpInterval="1000" maxEntries="10" /> > > <unsafe unreliableReturnValues="true" /> > > </namedCache> > </infinispan> > > > phil > > > > On Wed, Feb 3, 2010 at 6:42 PM, Manik Surtani <manik(a)jboss.org&gt; wrote: > >> Ugh, good point. I thought the unit tests would have trapped a dumb-ass >> mistake like this. >> >> The reason for transforming the name of the bucket is that we usually use >> hashcodes as the bucket name, which can take Integer.MIN_VALUE to >> Integer.MAX_VALUE. These are then translated into Strings, and this becomes >> the name of the storage unit, e.g., 12345.bucket in the FileCacheStore. Now >> filesystems are happy to accept a -12345.bucket but certain cloud storage >> providers barf when encountering the '-' character. Hence the >> transformation to A12345.bucket in some cases. >> >> Cheers >> Manik >> >> PS: pushing up a new snapshot as I type, containing this fix + lower >> verbosity on eviction-related lock timeouts. >> >> On 3 Feb 2010, at 17:16, Philippe Van Dyck wrote: >> >> And BTW, why do it ? >> >> p >> >> ---------- Forwarded message ---------- >> From: Philippe Van Dyck <pvdyck(a)gmail.com&gt; >> Date: Wed, Feb 3, 2010 at 6:15 PM >> Subject: CloudCacheStore Bug >> To: infinispan -Dev List <infinispan-dev(a)lists.jboss.org&gt; >> >> >> Hi all, >> >> there is a bug in CloudCacheStore that makes me feel like I am the only >> one using it ;-) >> >> in CR4 : if you change the "-" sign to "A" in getBucketName ... you need >> to do the opposite somewhere (or call it every time) ;-) >> >> WDYT ? >> >> p >> >> _______________________________________________ >> infinispan-dev mailing list >> >> infinispan-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> >> >> -- >> Manik Surtani >> manik(a)jboss.org >> Lead, Infinispan >> Lead, JBoss Cache >> http://www.infinispan.org >> http://www.jbosscache.org >> >> >> >> >> >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev > > > -- > Manik Surtani > manik(a)jboss.org > Lead, Infinispan > Lead, JBoss Cache > http://www.infinispan.org > http://www.jbosscache.org > > > > > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev > _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev -- Manik Surtani manik(a)jboss.org Lead, Infinispan Lead, JBoss Cache http://www.infinispan.org http://www.jbosscache.org _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

Am I missing something ? Loosing data is something I cannot afford ! I Plan to use this store as a *permanent* one... I have no backup ! (Actually S3 is the backup) - So, no, I don't want this ... at any price ;-) Then set <async enabled="false" /> in your cache store config. :-)

> reduced by looking through the async queue as well, before checking the > underlying store. But as I said, this just reduces the size of this window > and not eliminate it altogether, since this is async and there is no > guarantee that the cache store has finished writing internally (e.g., an > fsync() operation or in the case of S3, Amazon's eventual consistency > model). > > > Why should eviction be transactional? I don't need eviction to be an > all-or-nothing, reversible event. :) If an entry gets evicted, cool. If > not (for whatever reason), too bad, move on to the next evictable entry. > You are right, we don't want to rollback evictions... but maybe we should use a priority queue to be sure that evictions are done after any other command ? Doesn't it solve it all ? 1) The eviction thread runs (we could lower the priority of this thread too) 2) It fills a queue of keys to evict 3) The async queue is prioritized and evicts entries ... when there is nothing else to do (suddenly it looks like garbage collecting) That is a possibility. But I don't expect to be making any drastic changes to the existing eviction code anymore. Don't know if you have been following discussions re: LIRS, lock amortization, etc., but Vladimir is working on some very interesting self-evicting, bounded data containers which would mean that the eviction threads, etc all get ripped out.

On 4 Feb 2010, at 16:27, Philippe Van Dyck wrote: > Am I missing something ? Loosing data is something I cannot afford ! I > Plan to use this store as a *permanent* one... I have no backup ! (Actually > S3 is the backup) - So, no, I don't want this ... at any price ;-) > > > Then set <async enabled="false" /> in your cache store config. :-) > That is exactly what I planned to do... for the FileCacheStore since the latency is quite low and the failure rate almost zero. But the S3 store is very slow, and asynchronism is not a luxury... Right now, I am trying to make my own custom solution based on the size of the cache in memory (as trigger) and then I will evict specific oldest entries... hoping that async transactions are fully committed. > > > >> reduced by looking through the async queue as well, before checking the >> underlying store. But as I said, this just reduces the size of this window >> and not eliminate it altogether, since this is async and there is no >> guarantee that the cache store has finished writing internally (e.g., an >> fsync() operation or in the case of S3, Amazon's eventual consistency >> model). >> >> >> Why should eviction be transactional? I don't need eviction to be an >> all-or-nothing, reversible event. :) If an entry gets evicted, cool. If >> not (for whatever reason), too bad, move on to the next evictable entry. >> > > You are right, we don't want to rollback evictions... but maybe we should > use a priority queue to be sure that evictions are done after any other > command ? Doesn't it solve it all ? > > 1) The eviction thread runs (we could lower the priority of this thread > too) > 2) It fills a queue of keys to evict > 3) The async queue is prioritized and evicts entries ... when there is > nothing else to do (suddenly it looks like garbage collecting) > > > That is a possibility. But I don't expect to be making any drastic > changes to the existing eviction code anymore. Don't know if you have been > following discussions re: LIRS, lock amortization, etc., but Vladimir is > working on some very interesting self-evicting, bounded data containers > which would mean that the eviction threads, etc all get ripped out. > > Sounds terrific... Just to close the subject, shouldn't the documentation explicitly say that async and eviction are not "compatible" ? I don't think this really has anything to do with "incompatibilities". It's just the effects of queued/batched processing in the cache store async threads. You will see the same problem if you: 1. put (K, V) 2. The put is enqueued in the cache store 3. Restart the cache 4. get (K) // Data loss!? Just an async write that didn't have time to complete. And the above has nothing to do with eviction. Cheers Manik -- Manik Surtani manik(a)jboss.org Lead, Infinispan Lead, JBoss Cache http://www.infinispan.org http://www.jbosscache.org _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

Anyway, bad news, still have the problem without async and with purgeSynchronously="true". It is easy to test, create a transaction with 100000 updates (file store) and use a maxEntries of "2" for the eviction. While the transaction is being committed, the eviction thread wakes up and deletes entries. I don't think this behavior is intended (?) phil On Thu, Feb 4, 2010 at 6:19 PM, Philippe Van Dyck <pvdyck(a)gmail.com&gt;wrote: > Totally agree... as long as any failing async transaction is logged... > > BTW, since none of my cache entries expire, they are all instances of > ImmortalCacheEntry. > But since ImmortalCacheEntry does not update the "lastUsed" field, LRU or > FIFO are useless eviction strategies... > > My own eviction strategy, getting rid of a % of the size of the cache in > memory -> LRU first, does not work... > > Any idea ? Should I use my own timer (even if there is one in > InternalCacheEntry) ? > > cheers, > > phil > > On Thu, Feb 4, 2010 at 5:49 PM, Manik Surtani <manik(a)jboss.org&gt; wrote: > >> >> On 4 Feb 2010, at 16:27, Philippe Van Dyck wrote: >> >> >> >>> Am I missing something ? Loosing data is something I cannot afford ! I >>> Plan to use this store as a *permanent* one... I have no backup ! (Actually >>> S3 is the backup) - So, no, I don't want this ... at any price ;-) >>> >>> >>> Then set <async enabled="false" /> in your cache store config. :-) >>> >> >> That is exactly what I planned to do... for the FileCacheStore since the >> latency is quite low and the failure rate almost zero. >> But the S3 store is very slow, and asynchronism is not a luxury... >> >> Right now, I am trying to make my own custom solution based on the size >> of the cache in memory (as trigger) and then I will evict specific oldest >> entries... hoping that async transactions are fully committed. >> >> >>> >>> >>> >>>> reduced by looking through the async queue as well, before checking the >>>> underlying store. But as I said, this just reduces the size of this window >>>> and not eliminate it altogether, since this is async and there is no >>>> guarantee that the cache store has finished writing internally (e.g., an >>>> fsync() operation or in the case of S3, Amazon's eventual consistency >>>> model). >>>> >>>> >>>> Why should eviction be transactional? I don't need eviction to be an >>>> all-or-nothing, reversible event. :) If an entry gets evicted, cool. If >>>> not (for whatever reason), too bad, move on to the next evictable entry. >>>> >>> >>> You are right, we don't want to rollback evictions... but maybe we >>> should use a priority queue to be sure that evictions are done after any >>> other command ? Doesn't it solve it all ? >>> >>> 1) The eviction thread runs (we could lower the priority of this thread >>> too) >>> 2) It fills a queue of keys to evict >>> 3) The async queue is prioritized and evicts entries ... when there is >>> nothing else to do (suddenly it looks like garbage collecting) >>> >>> >>> That is a possibility. But I don't expect to be making any drastic >>> changes to the existing eviction code anymore. Don't know if you have been >>> following discussions re: LIRS, lock amortization, etc., but Vladimir is >>> working on some very interesting self-evicting, bounded data containers >>> which would mean that the eviction threads, etc all get ripped out. >>> >>> >> Sounds terrific... >> >> Just to close the subject, shouldn't the documentation explicitly say >> that async and eviction are not "compatible" ? >> >> >> I don't think this really has anything to do with "incompatibilities". >> It's just the effects of queued/batched processing in the cache store async >> threads. You will see the same problem if you: >> >> 1. put (K, V) >> 2. The put is enqueued in the cache store >> 3. Restart the cache >> 4. get (K) // Data loss!? Just an async write that didn't have time to >> complete. >> >> And the above has nothing to do with eviction. >> >> Cheers >> Manik >> -- >> Manik Surtani >> manik(a)jboss.org >> Lead, Infinispan >> Lead, JBoss Cache >> http://www.infinispan.org >> http://www.jbosscache.org >> >> >> >> >> >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> > >

Something else, the fix to the CloudCacheStore, it does not work. You should get rid of this "getBucketName" and simply add a single letter prefix to each bucketName (to avoid the minus sign which is valid in URLs BTW). And don't forget to use it everywhere a bucketName is used like i.e. readFromBlob or loadBucket and not only in insertBucket.

My I suppose that this code is not tested because you don't want to expose your S3 credentials ? And again, thanks for your help ;-) phil On Thu, Feb 4, 2010 at 6:36 PM, Philippe Van Dyck <pvdyck(a)gmail.com&gt; wrote: Anyway, bad news, still have the problem without async and with purgeSynchronously="true". It is easy to test, create a transaction with 100000 updates (file store) and use a maxEntries of "2" for the eviction. While the transaction is being committed, the eviction thread wakes up and deletes entries. I don't think this behavior is intended (?) phil On Thu, Feb 4, 2010 at 6:19 PM, Philippe Van Dyck <pvdyck(a)gmail.com&gt; wrote: Totally agree... as long as any failing async transaction is logged... BTW, since none of my cache entries expire, they are all instances of ImmortalCacheEntry. But since ImmortalCacheEntry does not update the "lastUsed" field, LRU or FIFO are useless eviction strategies... My own eviction strategy, getting rid of a % of the size of the cache in memory -> LRU first, does not work... Any idea ? Should I use my own timer (even if there is one in InternalCacheEntry) ? cheers, phil On Thu, Feb 4, 2010 at 5:49 PM, Manik Surtani <manik(a)jboss.org&gt; wrote: On 4 Feb 2010, at 16:27, Philippe Van Dyck wrote: > >> >> Am I missing something ? Loosing data is something I cannot afford ! I Plan to use this store as a *permanent* one... I have no backup ! (Actually S3 is the backup) - So, no, I don't want this ... at any price ;-) > > Then set <async enabled="false" /> in your cache store config. :-) > > That is exactly what I planned to do... for the FileCacheStore since the latency is quite low and the failure rate almost zero. > But the S3 store is very slow, and asynchronism is not a luxury... > > Right now, I am trying to make my own custom solution based on the size of the cache in memory (as trigger) and then I will evict specific oldest entries... hoping that async transactions are fully committed. > > >> >> reduced by looking through the async queue as well, before checking the underlying store. But as I said, this just reduces the size of this window and not eliminate it altogether, since this is async and there is no guarantee that the cache store has finished writing internally (e.g., an fsync() operation or in the case of S3, Amazon's eventual consistency model). >> >> >> Why should eviction be transactional? I don't need eviction to be an all-or-nothing, reversible event. :) If an entry gets evicted, cool. If not (for whatever reason), too bad, move on to the next evictable entry. >> >> You are right, we don't want to rollback evictions... but maybe we should use a priority queue to be sure that evictions are done after any other command ? Doesn't it solve it all ? >> >> 1) The eviction thread runs (we could lower the priority of this thread too) >> 2) It fills a queue of keys to evict >> 3) The async queue is prioritized and evicts entries ... when there is nothing else to do (suddenly it looks like garbage collecting) > > That is a possibility. But I don't expect to be making any drastic changes to the existing eviction code anymore. Don't know if you have been following discussions re: LIRS, lock amortization, etc., but Vladimir is working on some very interesting self-evicting, bounded data containers which would mean that the eviction threads, etc all get ripped out. > > > Sounds terrific... > > Just to close the subject, shouldn't the documentation explicitly say that async and eviction are not "compatible" ? I don't think this really has anything to do with "incompatibilities". It's just the effects of queued/batched processing in the cache store async threads. You will see the same problem if you: 1. put (K, V) 2. The put is enqueued in the cache store 3. Restart the cache 4. get (K) // Data loss!? Just an async write that didn't have time to complete. And the above has nothing to do with eviction. Cheers Manik -- Manik Surtani manik(a)jboss.org Lead, Infinispan Lead, JBoss Cache http://www.infinispan.org http://www.jbosscache.org _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

On 4 Feb 2010, at 17:36, Philippe Van Dyck wrote: Anyway, bad news, still have the problem without async and with purgeSynchronously="true". It is easy to test, create a transaction with 100000 updates (file store) and use a maxEntries of "2" for the eviction. While the transaction is being committed, the eviction thread wakes up and deletes entries. I don't think this behavior is intended (?) Right, similar to using an async queue (which in effect is a batch write to the store), a transaction too is a batch write to the store when the transaction completes. So when the tx does finish, writes are flushed. prior to that, concurrent threads not seeing the entry is expected since we don't support READ_UNCOMMITTED semantics. phil On Thu, Feb 4, 2010 at 6:19 PM, Philippe Van Dyck <pvdyck(a)gmail.com&gt;wrote: > Totally agree... as long as any failing async transaction is logged... > > BTW, since none of my cache entries expire, they are all instances of > ImmortalCacheEntry. > But since ImmortalCacheEntry does not update the "lastUsed" field, LRU or > FIFO are useless eviction strategies... > > My own eviction strategy, getting rid of a % of the size of the cache in > memory -> LRU first, does not work... > > Any idea ? Should I use my own timer (even if there is one in > InternalCacheEntry) ? > > cheers, > > phil > > On Thu, Feb 4, 2010 at 5:49 PM, Manik Surtani <manik(a)jboss.org&gt; wrote: > >> >> On 4 Feb 2010, at 16:27, Philippe Van Dyck wrote: >> >> >> >>> Am I missing something ? Loosing data is something I cannot afford ! I >>> Plan to use this store as a *permanent* one... I have no backup ! (Actually >>> S3 is the backup) - So, no, I don't want this ... at any price ;-) >>> >>> >>> Then set <async enabled="false" /> in your cache store config. :-) >>> >> >> That is exactly what I planned to do... for the FileCacheStore since the >> latency is quite low and the failure rate almost zero. >> But the S3 store is very slow, and asynchronism is not a luxury... >> >> Right now, I am trying to make my own custom solution based on the size >> of the cache in memory (as trigger) and then I will evict specific oldest >> entries... hoping that async transactions are fully committed. >> >> >>> >>> >>> >>>> reduced by looking through the async queue as well, before checking the >>>> underlying store. But as I said, this just reduces the size of this window >>>> and not eliminate it altogether, since this is async and there is no >>>> guarantee that the cache store has finished writing internally (e.g., an >>>> fsync() operation or in the case of S3, Amazon's eventual consistency >>>> model). >>>> >>>> >>>> Why should eviction be transactional? I don't need eviction to be an >>>> all-or-nothing, reversible event. :) If an entry gets evicted, cool. If >>>> not (for whatever reason), too bad, move on to the next evictable entry. >>>> >>> >>> You are right, we don't want to rollback evictions... but maybe we >>> should use a priority queue to be sure that evictions are done after any >>> other command ? Doesn't it solve it all ? >>> >>> 1) The eviction thread runs (we could lower the priority of this thread >>> too) >>> 2) It fills a queue of keys to evict >>> 3) The async queue is prioritized and evicts entries ... when there is >>> nothing else to do (suddenly it looks like garbage collecting) >>> >>> >>> That is a possibility. But I don't expect to be making any drastic >>> changes to the existing eviction code anymore. Don't know if you have been >>> following discussions re: LIRS, lock amortization, etc., but Vladimir is >>> working on some very interesting self-evicting, bounded data containers >>> which would mean that the eviction threads, etc all get ripped out. >>> >>> >> Sounds terrific... >> >> Just to close the subject, shouldn't the documentation explicitly say >> that async and eviction are not "compatible" ? >> >> >> I don't think this really has anything to do with "incompatibilities". >> It's just the effects of queued/batched processing in the cache store async >> threads. You will see the same problem if you: >> >> 1. put (K, V) >> 2. The put is enqueued in the cache store >> 3. Restart the cache >> 4. get (K) // Data loss!? Just an async write that didn't have time to >> complete. >> >> And the above has nothing to do with eviction. >> >> Cheers >> Manik >> -- >> Manik Surtani >> manik(a)jboss.org >> Lead, Infinispan >> Lead, JBoss Cache >> http://www.infinispan.org >> http://www.jbosscache.org >> >> >> >> >> >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> > > _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev -- Manik Surtani manik(a)jboss.org Lead, Infinispan Lead, JBoss Cache http://www.infinispan.org http://www.jbosscache.org _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

On 4 Feb 2010, at 18:39, Philippe Van Dyck wrote: > Eviction tries to acquire a lock on the key to send to the store, isn't this key locked by the ongoing committing transaction ? > Is the "Read uncommitted" isolation level related ? Aren't we talking about write locks - not read locks ? > Do you mean that before the transaction is completely flushed, and all the locks acquired, the eviction mechanism could jump in, acquire a lock and destroy an entry ? Hm, that's a good point. The tx should own the WL and the eviction thread should not be able to evict the entry in question. > > BTW, the InternalEntryFactory#createNewEntry() did the trick ;-) > > phil > > > On Thu, Feb 4, 2010 at 7:10 PM, Manik Surtani <manik(a)jboss.org&gt; wrote: > > On 4 Feb 2010, at 17:36, Philippe Van Dyck wrote: > >> Anyway, bad news, still have the problem without async and with purgeSynchronously="true". >> >> It is easy to test, create a transaction with 100000 updates (file store) and use a maxEntries of "2" for the eviction. >> >> While the transaction is being committed, the eviction thread wakes up and deletes entries. >> >> I don't think this behavior is intended (?) > > Right, similar to using an async queue (which in effect is a batch write to the store), a transaction too is a batch write to the store when the transaction completes. So when the tx does finish, writes are flushed. prior to that, concurrent threads not seeing the entry is expected since we don't support READ_UNCOMMITTED semantics. > >> >> phil >> >> On Thu, Feb 4, 2010 at 6:19 PM, Philippe Van Dyck <pvdyck(a)gmail.com&gt; wrote: >> Totally agree... as long as any failing async transaction is logged... >> >> BTW, since none of my cache entries expire, they are all instances of ImmortalCacheEntry. >> But since ImmortalCacheEntry does not update the "lastUsed" field, LRU or FIFO are useless eviction strategies... >> >> My own eviction strategy, getting rid of a % of the size of the cache in memory -> LRU first, does not work... >> >> Any idea ? Should I use my own timer (even if there is one in InternalCacheEntry) ? >> >> cheers, >> >> phil >> >> On Thu, Feb 4, 2010 at 5:49 PM, Manik Surtani <manik(a)jboss.org&gt; wrote: >> >> On 4 Feb 2010, at 16:27, Philippe Van Dyck wrote: >> >>> >>>> >>>> Am I missing something ? Loosing data is something I cannot afford ! I Plan to use this store as a *permanent* one... I have no backup ! (Actually S3 is the backup) - So, no, I don't want this ... at any price ;-) >>> >>> Then set <async enabled="false" /> in your cache store config. :-) >>> >>> That is exactly what I planned to do... for the FileCacheStore since the latency is quite low and the failure rate almost zero. >>> But the S3 store is very slow, and asynchronism is not a luxury... >>> >>> Right now, I am trying to make my own custom solution based on the size of the cache in memory (as trigger) and then I will evict specific oldest entries... hoping that async transactions are fully committed. >>> >>> >>>> >>>> reduced by looking through the async queue as well, before checking the underlying store. But as I said, this just reduces the size of this window and not eliminate it altogether, since this is async and there is no guarantee that the cache store has finished writing internally (e.g., an fsync() operation or in the case of S3, Amazon's eventual consistency model). >>>> >>>> >>>> Why should eviction be transactional? I don't need eviction to be an all-or-nothing, reversible event. :) If an entry gets evicted, cool. If not (for whatever reason), too bad, move on to the next evictable entry. >>>> >>>> You are right, we don't want to rollback evictions... but maybe we should use a priority queue to be sure that evictions are done after any other command ? Doesn't it solve it all ? >>>> >>>> 1) The eviction thread runs (we could lower the priority of this thread too) >>>> 2) It fills a queue of keys to evict >>>> 3) The async queue is prioritized and evicts entries ... when there is nothing else to do (suddenly it looks like garbage collecting) >>> >>> That is a possibility. But I don't expect to be making any drastic changes to the existing eviction code anymore. Don't know if you have been following discussions re: LIRS, lock amortization, etc., but Vladimir is working on some very interesting self-evicting, bounded data containers which would mean that the eviction threads, etc all get ripped out. >>> >>> >>> Sounds terrific... >>> >>> Just to close the subject, shouldn't the documentation explicitly say that async and eviction are not "compatible" ? >> >> I don't think this really has anything to do with "incompatibilities". It's just the effects of queued/batched processing in the cache store async threads. You will see the same problem if you: >> >> 1. put (K, V) >> 2. The put is enqueued in the cache store >> 3. Restart the cache >> 4. get (K) // Data loss!? Just an async write that didn't have time to complete. >> >> And the above has nothing to do with eviction. >> >> Cheers >> Manik >> -- >> Manik Surtani >> manik(a)jboss.org >> Lead, Infinispan >> Lead, JBoss Cache >> http://www.infinispan.org >> http://www.jbosscache.org >> >> >> >> >> >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> >> >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev > > > -- > Manik Surtani > manik(a)jboss.org > Lead, Infinispan > Lead, JBoss Cache > http://www.infinispan.org > http://www.jbosscache.org > > > > > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev -- Manik Surtani manik(a)jboss.org Lead, Infinispan Lead, JBoss Cache http://www.infinispan.org http://www.jbosscache.org _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

Shouldn't we consider this a bug ?

There is obviously something fishy in the locking mechanism, since I suppose that the idea of "eviction - I will not interfere" was related to the "no wait" option when asking the lock. WDYT ? cheers, phil Le 5 févr. 2010 à 23:17, Manik Surtani a écrit : > > On 4 Feb 2010, at 18:39, Philippe Van Dyck wrote: > >> Eviction tries to acquire a lock on the key to send to the store, isn't this key locked by the ongoing committing transaction ? >> Is the "Read uncommitted" isolation level related ? Aren't we talking about write locks - not read locks ? >> Do you mean that before the transaction is completely flushed, and all the locks acquired, the eviction mechanism could jump in, acquire a lock and destroy an entry ? > > Hm, that's a good point. The tx should own the WL and the eviction thread should not be able to evict the entry in question. > >> >> BTW, the InternalEntryFactory#createNewEntry() did the trick ;-) >> >> phil >> >> >> On Thu, Feb 4, 2010 at 7:10 PM, Manik Surtani <manik(a)jboss.org&gt; wrote: >> >> On 4 Feb 2010, at 17:36, Philippe Van Dyck wrote: >> >>> Anyway, bad news, still have the problem without async and with purgeSynchronously="true". >>> >>> It is easy to test, create a transaction with 100000 updates (file store) and use a maxEntries of "2" for the eviction. >>> >>> While the transaction is being committed, the eviction thread wakes up and deletes entries. >>> >>> I don't think this behavior is intended (?) >> >> Right, similar to using an async queue (which in effect is a batch write to the store), a transaction too is a batch write to the store when the transaction completes. So when the tx does finish, writes are flushed. prior to that, concurrent threads not seeing the entry is expected since we don't support READ_UNCOMMITTED semantics. >> >>> >>> phil >>> >>> On Thu, Feb 4, 2010 at 6:19 PM, Philippe Van Dyck <pvdyck(a)gmail.com&gt; wrote: >>> Totally agree... as long as any failing async transaction is logged... >>> >>> BTW, since none of my cache entries expire, they are all instances of ImmortalCacheEntry. >>> But since ImmortalCacheEntry does not update the "lastUsed" field, LRU or FIFO are useless eviction strategies... >>> >>> My own eviction strategy, getting rid of a % of the size of the cache in memory -> LRU first, does not work... >>> >>> Any idea ? Should I use my own timer (even if there is one in InternalCacheEntry) ? >>> >>> cheers, >>> >>> phil >>> >>> On Thu, Feb 4, 2010 at 5:49 PM, Manik Surtani <manik(a)jboss.org&gt; wrote: >>> >>> On 4 Feb 2010, at 16:27, Philippe Van Dyck wrote: >>> >>>> >>>>> >>>>> Am I missing something ? Loosing data is something I cannot afford ! I Plan to use this store as a *permanent* one... I have no backup ! (Actually S3 is the backup) - So, no, I don't want this ... at any price ;-) >>>> >>>> Then set <async enabled="false" /> in your cache store config. :-) >>>> >>>> That is exactly what I planned to do... for the FileCacheStore since the latency is quite low and the failure rate almost zero. >>>> But the S3 store is very slow, and asynchronism is not a luxury... >>>> >>>> Right now, I am trying to make my own custom solution based on the size of the cache in memory (as trigger) and then I will evict specific oldest entries... hoping that async transactions are fully committed. >>>> >>>> >>>>> >>>>> reduced by looking through the async queue as well, before checking the underlying store. But as I said, this just reduces the size of this window and not eliminate it altogether, since this is async and there is no guarantee that the cache store has finished writing internally (e.g., an fsync() operation or in the case of S3, Amazon's eventual consistency model). >>>>> >>>>> >>>>> Why should eviction be transactional? I don't need eviction to be an all-or-nothing, reversible event. :) If an entry gets evicted, cool. If not (for whatever reason), too bad, move on to the next evictable entry. >>>>> >>>>> You are right, we don't want to rollback evictions... but maybe we should use a priority queue to be sure that evictions are done after any other command ? Doesn't it solve it all ? >>>>> >>>>> 1) The eviction thread runs (we could lower the priority of this thread too) >>>>> 2) It fills a queue of keys to evict >>>>> 3) The async queue is prioritized and evicts entries ... when there is nothing else to do (suddenly it looks like garbage collecting) >>>> >>>> That is a possibility. But I don't expect to be making any drastic changes to the existing eviction code anymore. Don't know if you have been following discussions re: LIRS, lock amortization, etc., but Vladimir is working on some very interesting self-evicting, bounded data containers which would mean that the eviction threads, etc all get ripped out. >>>> >>>> >>>> Sounds terrific... >>>> >>>> Just to close the subject, shouldn't the documentation explicitly say that async and eviction are not "compatible" ? >>> >>> I don't think this really has anything to do with "incompatibilities". It's just the effects of queued/batched processing in the cache store async threads. You will see the same problem if you: >>> >>> 1. put (K, V) >>> 2. The put is enqueued in the cache store >>> 3. Restart the cache >>> 4. get (K) // Data loss!? Just an async write that didn't have time to complete. >>> >>> And the above has nothing to do with eviction. >>> >>> Cheers >>> Manik >>> -- >>> Manik Surtani >>> manik(a)jboss.org >>> Lead, Infinispan >>> Lead, JBoss Cache >>> http://www.infinispan.org >>> http://www.jbosscache.org >>> >>> >>> >>> >>> >>> _______________________________________________ >>> infinispan-dev mailing list >>> infinispan-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>> >>> >>> _______________________________________________ >>> infinispan-dev mailing list >>> infinispan-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> >> >> -- >> Manik Surtani >> manik(a)jboss.org >> Lead, Infinispan >> Lead, JBoss Cache >> http://www.infinispan.org >> http://www.jbosscache.org >> >> >> >> >> >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev > > -- > Manik Surtani > manik(a)jboss.org > Lead, Infinispan > Lead, JBoss Cache > http://www.infinispan.org > http://www.jbosscache.org > > > > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

Hi Manik, Still happening with the latest snapshot, but only when combining eviction, asynchronism (and the filecachestore for some latency, but your memory setup with 10ms eviction should work). phil P.S.: Sorry for not getting back to you, I've been sick for the past week, back on track now. Le 8 févr. 2010 à 13:52, Manik Surtani a écrit : > > On 6 Feb 2010, at 12:00, philippe van dyck wrote: > >> Shouldn't we consider this a bug ? > > Hmm, I can't seem to reproduce this. Followed your instructions below, see: > > http://fisheye.jboss.org/changelog/Infinispan/trunk?cs=1465 > > If you're on IRC at some point, ping me and lets chat abt this. > >> >> There is obviously something fishy in the locking mechanism, since I suppose that the idea of "eviction - I will not interfere" was related to the "no wait" option when asking the lock. >> >> WDYT ? >> >> cheers, >> >> phil >> >> >> >> Le 5 févr. 2010 à 23:17, Manik Surtani a écrit : >> >>> >>> On 4 Feb 2010, at 18:39, Philippe Van Dyck wrote: >>> >>>> Eviction tries to acquire a lock on the key to send to the store, isn't this key locked by the ongoing committing transaction ? >>>> Is the "Read uncommitted" isolation level related ? Aren't we talking about write locks - not read locks ? >>>> Do you mean that before the transaction is completely flushed, and all the locks acquired, the eviction mechanism could jump in, acquire a lock and destroy an entry ? >>> >>> Hm, that's a good point. The tx should own the WL and the eviction thread should not be able to evict the entry in question. >>> >>>> >>>> BTW, the InternalEntryFactory#createNewEntry() did the trick ;-) >>>> >>>> phil >>>> >>>> >>>> On Thu, Feb 4, 2010 at 7:10 PM, Manik Surtani <manik(a)jboss.org&gt; wrote: >>>> >>>> On 4 Feb 2010, at 17:36, Philippe Van Dyck wrote: >>>> >>>>> Anyway, bad news, still have the problem without async and with purgeSynchronously="true". >>>>> >>>>> It is easy to test, create a transaction with 100000 updates (file store) and use a maxEntries of "2" for the eviction. >>>>> >>>>> While the transaction is being committed, the eviction thread wakes up and deletes entries. >>>>> >>>>> I don't think this behavior is intended (?) >>>> >>>> Right, similar to using an async queue (which in effect is a batch write to the store), a transaction too is a batch write to the store when the transaction completes. So when the tx does finish, writes are flushed. prior to that, concurrent threads not seeing the entry is expected since we don't support READ_UNCOMMITTED semantics. >>>> >>>>> >>>>> phil >>>>> >>>>> On Thu, Feb 4, 2010 at 6:19 PM, Philippe Van Dyck <pvdyck(a)gmail.com&gt; wrote: >>>>> Totally agree... as long as any failing async transaction is logged... >>>>> >>>>> BTW, since none of my cache entries expire, they are all instances of ImmortalCacheEntry. >>>>> But since ImmortalCacheEntry does not update the "lastUsed" field, LRU or FIFO are useless eviction strategies... >>>>> >>>>> My own eviction strategy, getting rid of a % of the size of the cache in memory -> LRU first, does not work... >>>>> >>>>> Any idea ? Should I use my own timer (even if there is one in InternalCacheEntry) ? >>>>> >>>>> cheers, >>>>> >>>>> phil >>>>> >>>>> On Thu, Feb 4, 2010 at 5:49 PM, Manik Surtani <manik(a)jboss.org&gt; wrote: >>>>> >>>>> On 4 Feb 2010, at 16:27, Philippe Van Dyck wrote: >>>>> >>>>>> >>>>>>> >>>>>>> Am I missing something ? Loosing data is something I cannot afford ! I Plan to use this store as a *permanent* one... I have no backup ! (Actually S3 is the backup) - So, no, I don't want this ... at any price ;-) >>>>>> >>>>>> Then set <async enabled="false" /> in your cache store config. :-) >>>>>> >>>>>> That is exactly what I planned to do... for the FileCacheStore since the latency is quite low and the failure rate almost zero. >>>>>> But the S3 store is very slow, and asynchronism is not a luxury... >>>>>> >>>>>> Right now, I am trying to make my own custom solution based on the size of the cache in memory (as trigger) and then I will evict specific oldest entries... hoping that async transactions are fully committed. >>>>>> >>>>>> >>>>>>> >>>>>>> reduced by looking through the async queue as well, before checking the underlying store. But as I said, this just reduces the size of this window and not eliminate it altogether, since this is async and there is no guarantee that the cache store has finished writing internally (e.g., an fsync() operation or in the case of S3, Amazon's eventual consistency model). >>>>>>> >>>>>>> >>>>>>> Why should eviction be transactional? I don't need eviction to be an all-or-nothing, reversible event. :) If an entry gets evicted, cool. If not (for whatever reason), too bad, move on to the next evictable entry. >>>>>>> >>>>>>> You are right, we don't want to rollback evictions... but maybe we should use a priority queue to be sure that evictions are done after any other command ? Doesn't it solve it all ? >>>>>>> >>>>>>> 1) The eviction thread runs (we could lower the priority of this thread too) >>>>>>> 2) It fills a queue of keys to evict >>>>>>> 3) The async queue is prioritized and evicts entries ... when there is nothing else to do (suddenly it looks like garbage collecting) >>>>>> >>>>>> That is a possibility. But I don't expect to be making any drastic changes to the existing eviction code anymore. Don't know if you have been following discussions re: LIRS, lock amortization, etc., but Vladimir is working on some very interesting self-evicting, bounded data containers which would mean that the eviction threads, etc all get ripped out. >>>>>> >>>>>> >>>>>> Sounds terrific... >>>>>> >>>>>> Just to close the subject, shouldn't the documentation explicitly say that async and eviction are not "compatible" ? >>>>> >>>>> I don't think this really has anything to do with "incompatibilities". It's just the effects of queued/batched processing in the cache store async threads. You will see the same problem if you: >>>>> >>>>> 1. put (K, V) >>>>> 2. The put is enqueued in the cache store >>>>> 3. Restart the cache >>>>> 4. get (K) // Data loss!? Just an async write that didn't have time to complete. >>>>> >>>>> And the above has nothing to do with eviction. >>>>> >>>>> Cheers >>>>> Manik >>>>> -- >>>>> Manik Surtani >>>>> manik(a)jboss.org >>>>> Lead, Infinispan >>>>> Lead, JBoss Cache >>>>> http://www.infinispan.org >>>>> http://www.jbosscache.org >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> infinispan-dev mailing list >>>>> infinispan-dev(a)lists.jboss.org >>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>>> >>>>> >>>>> _______________________________________________ >>>>> infinispan-dev mailing list >>>>> infinispan-dev(a)lists.jboss.org >>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>> >>>> >>>> -- >>>> Manik Surtani >>>> manik(a)jboss.org >>>> Lead, Infinispan >>>> Lead, JBoss Cache >>>> http://www.infinispan.org >>>> http://www.jbosscache.org >>>> >>>> >>>> >>>> >>>> >>>> _______________________________________________ >>>> infinispan-dev mailing list >>>> infinispan-dev(a)lists.jboss.org >>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>> >>>> _______________________________________________ >>>> infinispan-dev mailing list >>>> infinispan-dev(a)lists.jboss.org >>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>> >>> -- >>> Manik Surtani >>> manik(a)jboss.org >>> Lead, Infinispan >>> Lead, JBoss Cache >>> http://www.infinispan.org >>> http://www.jbosscache.org >>> >>> >>> >>> >>> _______________________________________________ >>> infinispan-dev mailing list >>> infinispan-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev > > -- > Manik Surtani > manik(a)jboss.org > Lead, Infinispan > Lead, JBoss Cache > http://www.infinispan.org > http://www.jbosscache.org > > > > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

Just to be sure : 1) create transaction 2) commit transaction 2.1) get write locks for all entries contained in the transaction 2.2) write all cache entries 2.3) release all locks 2.4) commit returns 3) control returns to the calling thread (the async or the sync one) While a thread checks for deadlocks and another one tries to evict entries (using a 0ms timeout to avoid evicting locked ones). "Read committed" isolation level specifies that write locks must be released at the end of the transaction so ... if the eviction thread asks for a write lock, they should not interfere. I am obviously missing something... Should I understand that, when using asynchronism, I should not expect this behavior at all ? Actually, should I expect a big difference in processing, when using an asynch cache store ? I wonder if there should be one... Should I loose my transaction boundaries because I commit the transaction "later" ? WDYT ? phil

Le 16 févr. 2010 à 18:36, Manik Surtani a écrit : > Yes, with an async cache store this is possible since the entry may not have been written to the store by the time the reader thread looks for an entry that has already been evicted, transaction or no transaction. > > On 16 Feb 2010, at 15:57, philippe van dyck wrote: > >> Hi Manik, >> >> Still happening with the latest snapshot, but only when combining eviction, asynchronism (and the filecachestore for some latency, but your memory setup with 10ms eviction should work). >> >> phil >> >> P.S.: Sorry for not getting back to you, I've been sick for the past week, back on track now. >> >> Le 8 févr. 2010 à 13:52, Manik Surtani a écrit : >> >>> >>> On 6 Feb 2010, at 12:00, philippe van dyck wrote: >>> >>>> Shouldn't we consider this a bug ? >>> >>> Hmm, I can't seem to reproduce this. Followed your instructions below, see: >>> >>> http://fisheye.jboss.org/changelog/Infinispan/trunk?cs=1465 >>> >>> If you're on IRC at some point, ping me and lets chat abt this. >>> >>>> >>>> There is obviously something fishy in the locking mechanism, since I suppose that the idea of "eviction - I will not interfere" was related to the "no wait" option when asking the lock. >>>> >>>> WDYT ? >>>> >>>> cheers, >>>> >>>> phil >>>> >>>> >>>> >>>> Le 5 févr. 2010 à 23:17, Manik Surtani a écrit : >>>> >>>>> >>>>> On 4 Feb 2010, at 18:39, Philippe Van Dyck wrote: >>>>> >>>>>> Eviction tries to acquire a lock on the key to send to the store, isn't this key locked by the ongoing committing transaction ? >>>>>> Is the "Read uncommitted" isolation level related ? Aren't we talking about write locks - not read locks ? >>>>>> Do you mean that before the transaction is completely flushed, and all the locks acquired, the eviction mechanism could jump in, acquire a lock and destroy an entry ? >>>>> >>>>> Hm, that's a good point. The tx should own the WL and the eviction thread should not be able to evict the entry in question. >>>>> >>>>>> >>>>>> BTW, the InternalEntryFactory#createNewEntry() did the trick ;-) >>>>>> >>>>>> phil >>>>>> >>>>>> >>>>>> On Thu, Feb 4, 2010 at 7:10 PM, Manik Surtani <manik(a)jboss.org&gt; wrote: >>>>>> >>>>>> On 4 Feb 2010, at 17:36, Philippe Van Dyck wrote: >>>>>> >>>>>>> Anyway, bad news, still have the problem without async and with purgeSynchronously="true". >>>>>>> >>>>>>> It is easy to test, create a transaction with 100000 updates (file store) and use a maxEntries of "2" for the eviction. >>>>>>> >>>>>>> While the transaction is being committed, the eviction thread wakes up and deletes entries. >>>>>>> >>>>>>> I don't think this behavior is intended (?) >>>>>> >>>>>> Right, similar to using an async queue (which in effect is a batch write to the store), a transaction too is a batch write to the store when the transaction completes. So when the tx does finish, writes are flushed. prior to that, concurrent threads not seeing the entry is expected since we don't support READ_UNCOMMITTED semantics. >>>>>> >>>>>>> >>>>>>> phil >>>>>>> >>>>>>> On Thu, Feb 4, 2010 at 6:19 PM, Philippe Van Dyck <pvdyck(a)gmail.com&gt; wrote: >>>>>>> Totally agree... as long as any failing async transaction is logged... >>>>>>> >>>>>>> BTW, since none of my cache entries expire, they are all instances of ImmortalCacheEntry. >>>>>>> But since ImmortalCacheEntry does not update the "lastUsed" field, LRU or FIFO are useless eviction strategies... >>>>>>> >>>>>>> My own eviction strategy, getting rid of a % of the size of the cache in memory -> LRU first, does not work... >>>>>>> >>>>>>> Any idea ? Should I use my own timer (even if there is one in InternalCacheEntry) ? >>>>>>> >>>>>>> cheers, >>>>>>> >>>>>>> phil >>>>>>> >>>>>>> On Thu, Feb 4, 2010 at 5:49 PM, Manik Surtani <manik(a)jboss.org&gt; wrote: >>>>>>> >>>>>>> On 4 Feb 2010, at 16:27, Philippe Van Dyck wrote: >>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>> Am I missing something ? Loosing data is something I cannot afford ! I Plan to use this store as a *permanent* one... I have no backup ! (Actually S3 is the backup) - So, no, I don't want this ... at any price ;-) >>>>>>>> >>>>>>>> Then set <async enabled="false" /> in your cache store config. :-) >>>>>>>> >>>>>>>> That is exactly what I planned to do... for the FileCacheStore since the latency is quite low and the failure rate almost zero. >>>>>>>> But the S3 store is very slow, and asynchronism is not a luxury... >>>>>>>> >>>>>>>> Right now, I am trying to make my own custom solution based on the size of the cache in memory (as trigger) and then I will evict specific oldest entries... hoping that async transactions are fully committed. >>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>> reduced by looking through the async queue as well, before checking the underlying store. But as I said, this just reduces the size of this window and not eliminate it altogether, since this is async and there is no guarantee that the cache store has finished writing internally (e.g., an fsync() operation or in the case of S3, Amazon's eventual consistency model). >>>>>>>>> >>>>>>>>> >>>>>>>>> Why should eviction be transactional? I don't need eviction to be an all-or-nothing, reversible event. :) If an entry gets evicted, cool. If not (for whatever reason), too bad, move on to the next evictable entry. >>>>>>>>> >>>>>>>>> You are right, we don't want to rollback evictions... but maybe we should use a priority queue to be sure that evictions are done after any other command ? Doesn't it solve it all ? >>>>>>>>> >>>>>>>>> 1) The eviction thread runs (we could lower the priority of this thread too) >>>>>>>>> 2) It fills a queue of keys to evict >>>>>>>>> 3) The async queue is prioritized and evicts entries ... when there is nothing else to do (suddenly it looks like garbage collecting) >>>>>>>> >>>>>>>> That is a possibility. But I don't expect to be making any drastic changes to the existing eviction code anymore. Don't know if you have been following discussions re: LIRS, lock amortization, etc., but Vladimir is working on some very interesting self-evicting, bounded data containers which would mean that the eviction threads, etc all get ripped out. >>>>>>>> >>>>>>>> >>>>>>>> Sounds terrific... >>>>>>>> >>>>>>>> Just to close the subject, shouldn't the documentation explicitly say that async and eviction are not "compatible" ? >>>>>>> >>>>>>> I don't think this really has anything to do with "incompatibilities". It's just the effects of queued/batched processing in the cache store async threads. You will see the same problem if you: >>>>>>> >>>>>>> 1. put (K, V) >>>>>>> 2. The put is enqueued in the cache store >>>>>>> 3. Restart the cache >>>>>>> 4. get (K) // Data loss!? Just an async write that didn't have time to complete. >>>>>>> >>>>>>> And the above has nothing to do with eviction. >>>>>>> >>>>>>> Cheers >>>>>>> Manik >>>>>>> -- >>>>>>> Manik Surtani >>>>>>> manik(a)jboss.org >>>>>>> Lead, Infinispan >>>>>>> Lead, JBoss Cache >>>>>>> http://www.infinispan.org >>>>>>> http://www.jbosscache.org >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> infinispan-dev mailing list >>>>>>> infinispan-dev(a)lists.jboss.org >>>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> infinispan-dev mailing list >>>>>>> infinispan-dev(a)lists.jboss.org >>>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>>>> >>>>>> >>>>>> -- >>>>>> Manik Surtani >>>>>> manik(a)jboss.org >>>>>> Lead, Infinispan >>>>>> Lead, JBoss Cache >>>>>> http://www.infinispan.org >>>>>> http://www.jbosscache.org >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> infinispan-dev mailing list >>>>>> infinispan-dev(a)lists.jboss.org >>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>>>> >>>>>> _______________________________________________ >>>>>> infinispan-dev mailing list >>>>>> infinispan-dev(a)lists.jboss.org >>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>>> >>>>> -- >>>>> Manik Surtani >>>>> manik(a)jboss.org >>>>> Lead, Infinispan >>>>> Lead, JBoss Cache >>>>> http://www.infinispan.org >>>>> http://www.jbosscache.org >>>>> >>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> infinispan-dev mailing list >>>>> infinispan-dev(a)lists.jboss.org >>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>> >>>> _______________________________________________ >>>> infinispan-dev mailing list >>>> infinispan-dev(a)lists.jboss.org >>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>> >>> -- >>> Manik Surtani >>> manik(a)jboss.org >>> Lead, Infinispan >>> Lead, JBoss Cache >>> http://www.infinispan.org >>> http://www.jbosscache.org >>> >>> >>> >>> >>> _______________________________________________ >>> infinispan-dev mailing list >>> infinispan-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev > > -- > Manik Surtani > manik(a)jboss.org > Lead, Infinispan > Lead, JBoss Cache > http://www.infinispan.org > http://www.jbosscache.org > > > > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev