[infinispan-dev] Fwd: CloudCacheStore Bug

Thu Feb 4 13:39:39 EST 2010

Eviction tries to acquire a lock on the key to send to the store, isn't this
key locked by the ongoing committing transaction ?
Is the "Read uncommitted" isolation level related ? Aren't we talking about
write locks - not read locks ?
Do you mean that before the transaction is completely flushed, and all the
locks acquired, the eviction mechanism could jump in, acquire a lock and
destroy an entry ?

BTW, the InternalEntryFactory#createNewEntry() did the trick ;-)

phil

On Thu, Feb 4, 2010 at 7:10 PM, Manik Surtani <manik at jboss.org> wrote:

>
> On 4 Feb 2010, at 17:36, Philippe Van Dyck wrote:
>
> Anyway, bad news, still have the problem without async and
> with purgeSynchronously="true".
>
> It is easy to test, create a transaction with 100000 updates (file store)
> and use a maxEntries of "2" for the eviction.
>
> While the transaction is being committed, the eviction thread wakes up and
> deletes entries.
>
> I don't think this behavior is intended (?)
>
>
> Right, similar to using an async queue (which in effect is a batch write to
> the store), a transaction too is a batch write to the store when the
> transaction completes.  So when the tx does finish, writes are flushed.
>  prior to that, concurrent threads not seeing the entry is expected since we
> don't support READ_UNCOMMITTED semantics.
>
>
> phil
>
> On Thu, Feb 4, 2010 at 6:19 PM, Philippe Van Dyck <pvdyck at gmail.com>wrote:
>
>> Totally agree... as long as any failing async transaction is logged...
>>
>> BTW, since none of my cache entries expire, they are all instances of
>> ImmortalCacheEntry.
>> But since ImmortalCacheEntry does not update the "lastUsed" field, LRU or
>> FIFO are useless eviction strategies...
>>
>> My own eviction strategy, getting rid of a % of the size of the cache in
>> memory -> LRU first, does not work...
>>
>> Any idea ? Should I use my own timer (even if there is one in
>> InternalCacheEntry) ?
>>
>> cheers,
>>
>> phil
>>
>> On Thu, Feb 4, 2010 at 5:49 PM, Manik Surtani <manik at jboss.org> wrote:
>>
>>>
>>> On 4 Feb 2010, at 16:27, Philippe Van Dyck wrote:
>>>
>>>
>>>
>>>> Am I missing something ? Loosing data is something I cannot afford ! I
>>>> Plan to use this store as a *permanent* one... I have no backup ! (Actually
>>>> S3 is the backup) - So, no, I don't want this ... at any price ;-)
>>>>
>>>>
>>>> Then set <async enabled="false" /> in your cache store config.  :-)
>>>>
>>>
>>> That is exactly what I planned to do... for the FileCacheStore since the
>>> latency is quite low and the failure rate almost zero.
>>> But the S3 store is very slow, and asynchronism is not a luxury...
>>>
>>> Right now, I am trying to make my own custom solution based on the size
>>> of the cache in memory (as trigger) and then I will evict specific oldest
>>> entries... hoping that async transactions are fully committed.
>>>
>>>
>>>>
>>>>
>>>>
>>>>> reduced by looking through the async queue as well, before checking the
>>>>> underlying store.  But as I said, this just reduces the size of this window
>>>>> and not eliminate it altogether, since this is async and there is no
>>>>> guarantee that the cache store has finished writing internally (e.g., an
>>>>> fsync() operation or in the case of S3, Amazon's eventual consistency
>>>>> model).
>>>>>
>>>>>
>>>>> Why should eviction be transactional?  I don't need eviction to be an
>>>>> all-or-nothing, reversible event. :)  If an entry gets evicted, cool.  If
>>>>> not (for whatever reason), too bad, move on to the next evictable entry.
>>>>>
>>>>
>>>> You are right, we don't want to rollback evictions... but maybe we
>>>> should use a priority queue to be sure that evictions are done after any
>>>> other command ? Doesn't it solve it all ?
>>>>
>>>> 1) The eviction thread runs (we could lower the priority of this thread
>>>> too)
>>>> 2) It fills a queue of keys to evict
>>>> 3) The async queue is prioritized and evicts entries ... when there is
>>>> nothing else to do (suddenly it looks like garbage collecting)
>>>>
>>>>
>>>> That is a possibility.  But I don't expect to be making any drastic
>>>> changes to the existing eviction code anymore.  Don't know if you have been
>>>> following discussions re: LIRS, lock amortization, etc., but Vladimir is
>>>> working on some very interesting self-evicting, bounded data containers
>>>> which would mean that the eviction threads, etc all get ripped out.
>>>>
>>>>
>>> Sounds terrific...
>>>
>>> Just to close the subject, shouldn't the documentation explicitly say
>>> that async and eviction are not "compatible" ?
>>>
>>>
>>> I don't think this really has anything to do with "incompatibilities".
>>>  It's just the effects of queued/batched processing in the cache store async
>>> threads.  You will see the same problem if you:
>>>
>>> 1.  put (K, V)
>>> 2.  The put is enqueued in the cache store
>>> 3.  Restart the cache
>>> 4.  get (K) // Data loss!?  Just an async write that didn't have time to
>>> complete.
>>>
>>> And the above has nothing to do with eviction.
>>>
>>> Cheers
>>> Manik
>>>  --
>>> Manik Surtani
>>> manik at jboss.org
>>> Lead, Infinispan
>>> Lead, JBoss Cache
>>> http://www.infinispan.org
>>> http://www.jbosscache.org
>>>
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> infinispan-dev mailing list
>>> infinispan-dev at lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>
>>
>>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
>
> --
> Manik Surtani
> manik at jboss.org
> Lead, Infinispan
> Lead, JBoss Cache
> http://www.infinispan.org
> http://www.jbosscache.org
>
>
>
>
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/infinispan-dev/attachments/20100204/d86bc0fb/attachment-0002.html 