From: "Galder Zamarreño" <galder(a)redhat.com>
To: "Andrig Miller" <anmiller(a)redhat.com>
Cc: "Steve Ebersole" <steve(a)hibernate.org>, "John O'Hara"
<johara(a)redhat.com>, "Jeremy Whiting"
<jwhiting(a)redhat.com>, "infinispan -Dev List"
<infinispan-dev(a)lists.jboss.org>
Sent: Thursday, September 20, 2012 6:48:59 AM
Subject: Re: [infinispan-dev] Issue with cache blocks for local read-only cache
On Sep 19, 2012, at 4:20 PM, Andrig Miller <anmiller(a)redhat.com>
wrote:
> Yes, I can see how that can happen, if the data is deleted from
> outside the application.
^ The issue does not only happen if the data is deleted outside the
application. As indicated in
https://hibernate.onjira.com/browse/HHH-3817, this can happen with
two competing transactions.
> If you cache something as READ_ONLY, and it gets deleted, that
> doesn't fit the definition of READ_ONLY though. You are using the
> wrong cache concurrency strategy.
>
> Even that issue outlines the scenario where the collection is
> updated, which means its not a READ_ONLY.
I think the update is irrelevant here. The issue is related to
putFromLoad + remove, which both AFAIK, are allowed in READ_ONLY
(remember that we had the discussion on whether remove should be
allowed in a READ_ONLY cache:
https://hibernate.onjira.com/browse/HHH-7350).
Yes, remove can be done, its just update that matters to READ_ONLY. One thing I thought
about was I thought we were using MVCC for this stuff. Any transaction that reads from
the cache, while something is being added/removed, should be reading the read consistent
image, and should never wait on a lock, correct? We see all the threads in our thread
pool sitting in a blocked state based on this locking.
That really shouldn't be happening with MVCC.
Andy
>
> Andy
>
> ----- Original Message -----
>> From: "Galder Zamarreño" <galder(a)redhat.com>
>> To: "infinispan -Dev List" <infinispan-dev(a)lists.jboss.org>
>> Cc: "Steve Ebersole" <steve(a)hibernate.org>, "John
O'Hara"
>> <johara(a)redhat.com>, "Andrig Miller"
<anmiller(a)redhat.com>,
>> "Jeremy Whiting" <jwhiting(a)redhat.com>
>> Sent: Wednesday, September 19, 2012 2:48:37 AM
>> Subject: Re: [infinispan-dev] Issue with cache blocks for local
>> read-only cache
>>
>> This is code written in JBoss Cache days to deal with situations
>> where putFromLoad might try to store stale data (if data is
>> deleted
>> in between a database read and putFromLoad being called).
>>
>> Indeed this can happen with read only data, and has nothing to do
>> with clustering.
>>
>> The original issue is:
>>
https://hibernate.onjira.com/browse/HHH-3817
>>
>> I'll check how this can be improved.
>>
>> Cheers,
>>
>> On Sep 18, 2012, at 4:55 PM, Sanne Grinovero
>> <sanne(a)infinispan.org>
>> wrote:
>>
>>> There seems to be a single (global) reentrant lock which is
>>> acquired
>>> on any put; I don't know much about the module design, but there
>>> is
>>> a
>>> comment close to the lock acquisition mentioning a need to flush
>>> an
>>> operations queue to avoid a deadlock.
>>> Could it just make sure to reorder keys, so that deadlocks are
>>> avoided?
>>>
>>> On 18 September 2012 16:26, Manik Surtani <manik(a)jboss.org>
>>> wrote:
>>>> Agreed.
>>>>
>>>> This locking appears to be in the 2LC integration code to guard
>>>> a
>>>> local
>>>> collection though - there must be better, non-blocking ways to
>>>> guard this -
>>>> if it is needed at all for RO entities.
>>>>
>>>> - M
>>>>
>>>> On 18 Sep 2012, at 15:18, Andrig Miller <anmiller(a)redhat.com>
>>>> wrote:
>>>>
>>>> What I find interesting, is that in this use case, READ_ONLY
>>>> concurrency
>>>> strategy and a local cache (no invalidation and no replication),
>>>> there is no
>>>> need to do any locking.
>>>>
>>>> If we can get to a solution without any locking that would be
>>>> ideal.
>>>>
>>>> Andy
>>>>
>>>> ________________________________
>>>>
>>>> From: "Manik Surtani" <manik(a)jboss.org>
>>>> To: "Ståle W. Pedersen" <spederse(a)redhat.com>
>>>> Cc: "Galder Zamarreño" <galder(a)redhat.com>, "John
O'Hara"
>>>> <johara(a)redhat.com>, "Jeremy Whiting"
<jwhiting(a)redhat.com>,
>>>> "Andrig Miller"
>>>> <anmiller(a)redhat.com>, "Steve Ebersole"
<steve(a)hibernate.org>,
>>>> "infinispan-dev" <infinispan-dev(a)lists.jboss.org>
>>>> Sent: Tuesday, September 18, 2012 8:13:43 AM
>>>> Subject: Re: Issue with cache blocks for local read-only cache
>>>>
>>>>
>>>> Looking at your profiler snapshot, these locks are in the
>>>> Hibernate 2nd
>>>> level cache implementation for Infinispan. Galder, any ideas?
>>>>
>>>> - M
>>>>
>>>> On 18 Sep 2012, at 13:43, Ståle W. Pedersen
>>>> <spederse(a)redhat.com>
>>>> wrote:
>>>>
>>>> hi galder and manik, sorry for sending this mail to so many, but
>>>> we've ran
>>>> into a issue that prevents us from further scaling of the
>>>> specjenterprise2010 benchmark.
>>>>
>>>> so when doing specjenterprise2010 benchmark testing we've seen a
>>>> lot of
>>>> blocks caused by the entity/query cache. we've been testing with
>>>> only
>>>> caching a simple entity bean that's read-only and queries
>>>> related
>>>> to this
>>>> entity (selects).
>>>>
>>>> here is a screenshot of the hotspot
>>>>
found:https://dl.dropbox.com/u/30971563/specjent_block.png
>>>>
>>>> here is the standalone.xml:
>>>>
https://dl.dropbox.com/u/30971563/standalone-full.xml
>>>>
>>>> here is the orm.xml:
>>>>
https://dl.dropbox.com/u/30971563/order_orm.xml
>>>>
>>>> what we don't understand is why there are so many puts into the
>>>> cache for an
>>>> object that is marked as read-only. when we're testing without
>>>> caching we do
>>>> not see any blocks.
>>>>
>>>> any help/ideas would be great. if anyone want a jprofiler
>>>> snapshot
>>>> of the
>>>> run, let me know.
>>>>
>>>> regards, ståle
>>>> --
>>>> JBoss Performance Team Lead
>>>> JBoss by Red Hat
>>>>
>>>>
>>>> --
>>>> Manik Surtani
>>>> manik(a)jboss.org
>>>>
twitter.com/maniksurtani
>>>>
>>>> Platform Architect, JBoss Data Grid
>>>>
http://red.ht/data-grid
>>>>
>>>>
>>>> --
>>>> Manik Surtani
>>>> manik(a)jboss.org
>>>>
twitter.com/maniksurtani
>>>>
>>>> Platform Architect, JBoss Data Grid
>>>>
http://red.ht/data-grid
>>>>
>>>>
>>>> _______________________________________________
>>>> infinispan-dev mailing list
>>>> infinispan-dev(a)lists.jboss.org
>>>>
https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>
>>> _______________________________________________
>>> infinispan-dev mailing list
>>> infinispan-dev(a)lists.jboss.org
>>>
https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>
>>
>> --
>> Galder Zamarreño
>> galder(a)redhat.com
>>
twitter.com/galderz
>>
>> Project Lead, Escalante
>>
http://escalante.io
>>
>> Engineer, Infinispan
>>
http://infinispan.org
>>
>>
--
Galder Zamarreño
galder(a)redhat.com
twitter.com/galderz
Project Lead, Escalante
http://escalante.io
Engineer, Infinispan
http://infinispan.org