[infinispan-dev] [ISPN-116] Async cache store: aggregation of multiple changes on a single key

Tue Aug 4 05:31:13 EDT 2009

On 08/04/2009 11:26 AM, Galder Zamarreno wrote:
>
>
> On 08/04/2009 09:56 AM, Galder Zamarreno wrote:
>> Hi Amin,
>>
>
> </snip>
>
>>> 2. Using xml config, no storeLockSafe is called at all?
>
> That's because you have singleton store enabled but the cache is not
> configured with correctly to form a cluster. Either: set transport class
> correctly like this:
>
> <transport clusterName="SmppGatewayCluster"
> transportClass="org.infinispan.remoting.transport.jgroups.JGroupsTransport"/>
>
>
> or disable singleton store:
>
> <singletonStore enabled="false" pushStateWhenCoordinator="true"
> pushStateTimeout="20000"/>
>
> Note that enabling singleton store only makes sense when there's a
> cluster. I'll add a JIRA so that even if there isn't a cluster, having
> singleton store enabled does not stop you from storing stuff to cache
> store.

Actually, that should be a configuration error. Enabling singleton store 
does not make in a local cache or cache with no transport.

>
>>
>> I'll check this asap.
>>
>>> Thanks for your help,
>>> Amin Abbaspour
>>>
>>> On Mon, Jul 20, 2009 at 7:09 PM, Galder
>>> Zamarreno<galder.zamarreno at redhat.com> wrote:
>>>> On 07/20/2009 02:39 PM, Manik Surtani wrote:
>>>>> On 20 Jul 2009, at 12:35, Amin Abbaspour wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> Thanks Galder. I will test CoalescedAsyncStore ASAP and let you know.
>>>>>>
>>>>>> BTW Manik, I think the change is so much that we'd better see this as
>>>>>> a completely new store and later replace AsyncCacheStore or keep them
>>>>>> both.
>>>>> I agree that the impl is significantly different, but from a
>>>>> functional/feature standpoint, what does it offer over and above the
>>>>> AsyncCacheStore, except that it just does the same thing "better" ?
>>>>> :-)
>>>> I agree with Manik. I would have only left the other implementation
>>>> if there
>>>> was anything in particular the past implementation offered versus the
>>>> coalesced one. My perf tests indicated that we're as fast as before
>>>> but with
>>>> the added bonus of lesser cache store calls, so that looks to me a
>>>> win/win
>>>> situation.
>>>>
>>>> I've now committed https://jira.jboss.org/jira/browse/ISPN-116 and
>>>> added an
>>>> FAQ entry to
>>>> http://www.jboss.org/community/wiki/InfinispanTechnicalFAQs
>>>>
>>>> I'll talk about this further on a blog entry later this week.
>>>>
>>>> FAO Amin, if you wanna try it out, checkout latest from trunk and
>>>> let us
>>>> know what you think. No need for configuration changes, simply
>>>> add<async
>>>> enabled="true" /> and that should be enough.
>>>>
>>>>>> Regards,
>>>>>> Amin Abbaspour
>>>>>>
>>>>>> On Mon, Jul 20, 2009 at 2:47 PM, Manik Surtani<manik at jboss.org>
>>>>>> wrote:
>>>>>>> Nope, never got the email below.
>>>>>>>
>>>>>>> Anyway, I always thought the intention was to enhance or improve the
>>>>>>> AsyncCacheStore, not to create another one.
>>>>>>>
>>>>>>> Cheers
>>>>>>> Manik
>>>>>>>
>>>>>>> On 20 Jul 2009, at 10:55, Galder Zamarreno wrote:
>>>>>>>
>>>>>>>> Did other list readers ever got this message?
>>>>>>>>
>>>>>>>> I did send it last week but never actually got it via the dev
>>>>>>>> list, so
>>>>>>>> double checking...
>>>>>>>>
>>>>>>>> On 07/15/2009 07:58 PM, Galder Zamarreno wrote:
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> As part of this JIRA, I'm thinking whether we wanna leave the
>>>>>>>>> current
>>>>>>>>> queue based async store.
>>>>>>>>>
>>>>>>>>> For the work I've done, I've created a new class called
>>>>>>>>> CoalescedAsyncStore that works in the way agreed below, iow,
>>>>>>>>> instead of
>>>>>>>>> queying mods, it keeps a ConcurrentMap and which is swapped by a
>>>>>>>>> new
>>>>>>>>> instance when the async thread is going to apply the changes.
>>>>>>>>>
>>>>>>>>> I've run some perf tests between CoalescedAsyncStore and
>>>>>>>>> AsyncStore
>>>>>>>>> and
>>>>>>>>> when doing 1 million store where each key is different, the
>>>>>>>>> performance
>>>>>>>>> is fairly similar. The difference comes when trying to run a
>>>>>>>>> similar
>>>>>>>>> test where the same key is always updated, this results in lesser
>>>>>>>>> underlying store calls and hence CoalescedAsyncStore solution is
>>>>>>>>> faster
>>>>>>>>> here.
>>>>>>>>>
>>>>>>>>> So, rather than keeping both, I'd suggest replacing AsyncStore
>>>>>>>>> with
>>>>>>>>> the
>>>>>>>>> impl in CoalescedAsyncStore. We also need to look at the
>>>>>>>>> configuration
>>>>>>>>> for the new AsyncStore:
>>>>>>>>>
>>>>>>>>> Configuration wise in AsyncStoreConfig, I'd leave threadPoolSize,
>>>>>>>>> enabled and add the read/write lock acquisition timeout. The rest
>>>>>>>>> I'd
>>>>>>>>> get rid as no longer apply:
>>>>>>>>>
>>>>>>>>> int batchSize = 100;
>>>>>>>>> long pollWait = 100;
>>>>>>>>> int queueSize = 10000;
>>>>>>>>>
>>>>>>>>> By default, I'd give 5000ms to the read/write lock acquisition
>>>>>>>>> timeout.
>>>>>>>>>
>>>>>>>>> Thoughts?
>>>>>>>>>
>>>>>>>>> On 07/09/2009 01:08 PM, Manik Surtani wrote:
>>>>>>>>>> On 9 Jul 2009, at 11:37, Galder Zamarreno wrote:
>>>>>>>>>>
>>>>>>>>>>> On 07/09/2009 11:55 AM, Manik Surtani wrote:
>>>>>>>>>>>> On 8 Jul 2009, at 19:53, Jason T. Greene wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Manik Surtani wrote:
>>>>>>>>>>>>>> * Make the concurrent map volatile
>>>>>>>>>>>>>> * When iterating, first create a new ConcurrentMap and
>>>>>>>>>>>>>> replace
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>> old one with the new one so all concurrent threads write to
>>>>>>>>>>>>>> the new
>>>>>>>>>>>>>> Map
>>>>>>>>>>>>>> * Iterate over the old map
>>>>>>>>>>>>> That would lead to race conditions since a concurrent writing
>>>>>>>>>>>>> thread
>>>>>>>>>>>>> could write to the "old" map, either by getting a recent yet
>>>>>>>>>>>>> incorrect
>>>>>>>>>>>>> read off the volatile, or reading it right before it changes.
>>>>>>>>>>>> True, since referencing the map and writing to it isn't atomic.
>>>>>>>>>>>>
>>>>>>>>>>>> We could guard access to the map with a read/write lock. Safe,
>>>>>>>>>>>> if a
>>>>>>>>>>>> little heavy-handed... map writers would acquire a RL (since we
>>>>>>>>>>>> want
>>>>>>>>>>>> concurrent access here) but the async flushing thread would
>>>>>>>>>>>> need
>>>>>>>>>>>> to
>>>>>>>>>>>> wait
>>>>>>>>>>>> for a WL to swap the map reference, releasing the lock after
>>>>>>>>>>>> the
>>>>>>>>>>>> map
>>>>>>>>>>>> reference has been swapped.
>>>>>>>>>>> Yeah, I don't think it could be volatile because it only
>>>>>>>>>>> applies to
>>>>>>>>>>> the variable itself, not the contents pointed at. The R/W lock
>>>>>>>>>>> approach looks like a valid one and better than using
>>>>>>>>>>> synchronized
>>>>>>>>>>> blocks. Another thing I had in my mind is whether we need the
>>>>>>>>>>> copy to
>>>>>>>>>>> be a ConcurrentMap, we could probably just use a
>>>>>>>>>>> Immutables.immutableMapCopy().
>>>>>>>>>> Sync blocks would suck since it is effectively an exclusive
>>>>>>>>>> lock. We
>>>>>>>>>> want non-exclusive (read) locks for threads writing to this map
>>>>>>>>>> - we
>>>>>>>>>> want multiple application threads to do this concurrently
>>>>>>>>>> otherwise this
>>>>>>>>>> becomes a bottleneck. And hence the need for a ConcurrentMap.
>>>>>>>>>>
>>>>>>>>>> And we don't make a copy at all - all the async flushing thread
>>>>>>>>>> does is
>>>>>>>>>> that it creates a new, empty ConcurrentMap and swaps the 'async
>>>>>>>>>> queue'
>>>>>>>>>> Map so that application threads can continue logging their
>>>>>>>>>> changes
>>>>>>>>>> somewhere while old changes can be iterated through and flushed.
>>>>>>>>>> And for
>>>>>>>>>> this, the field still needs to be volatile (not strictly so,
>>>>>>>>>> since
>>>>>>>>>> we
>>>>>>>>>> can rely on the fencing that will happen in the lock as a side-
>>>>>>>>>> effect)
>>>>>>>>>>
>>>>>>>>>>> I looked at the two implementations (ConcurrentHashMap(Map)
>>>>>>>>>>> constructor vs Immutables.immutableMapCopy() which would
>>>>>>>>>>> Object.clone() on the map) but I'm not sure which one would be
>>>>>>>>>>> faster.
>>>>>>>>>>> Since clone() would rely on a native clone() impl, I'd imagine
>>>>>>>>>>> that to
>>>>>>>>>>> be faster.
>>>>>>>>>> Like I said, we don't need a copy. We need a new, empty map for
>>>>>>>>>> threads
>>>>>>>>>> to log changes to while the old one's being flushed.
>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Manik Surtani
>>>>>>>>>>>> manik at jboss.org
>>>>>>>>>>>> Lead, Infinispan
>>>>>>>>>>>> Lead, JBoss Cache
>>>>>>>>>>>> http://www.infinispan.org
>>>>>>>>>>>> http://www.jbosscache.org
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>> infinispan-dev mailing list
>>>>>>>>>>>> infinispan-dev at lists.jboss.org
>>>>>>>>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>>>>>>>>> --
>>>>>>>>>>> Galder Zamarreño
>>>>>>>>>>> Sr. Software Engineer
>>>>>>>>>>> Infinispan, JBoss Cache
>>>>>>>>>> --
>>>>>>>>>> Manik Surtani
>>>>>>>>>> manik at jboss.org
>>>>>>>>>> Lead, Infinispan
>>>>>>>>>> Lead, JBoss Cache
>>>>>>>>>> http://www.infinispan.org
>>>>>>>>>> http://www.jbosscache.org
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>> --
>>>>>>>> Galder Zamarreño
>>>>>>>> Sr. Software Engineer
>>>>>>>> Infinispan, JBoss Cache
>>>>>>>> _______________________________________________
>>>>>>>> infinispan-dev mailing list
>>>>>>>> infinispan-dev at lists.jboss.org
>>>>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>>>>> --
>>>>>>> Manik Surtani
>>>>>>> manik at jboss.org
>>>>>>> Lead, Infinispan
>>>>>>> Lead, JBoss Cache
>>>>>>> http://www.infinispan.org
>>>>>>> http://www.jbosscache.org
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> infinispan-dev mailing list
>>>>>>> infinispan-dev at lists.jboss.org
>>>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>>>>>
>>>>>> _______________________________________________
>>>>>> infinispan-dev mailing list
>>>>>> infinispan-dev at lists.jboss.org
>>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>>> --
>>>>> Manik Surtani
>>>>> manik at jboss.org
>>>>> Lead, Infinispan
>>>>> Lead, JBoss Cache
>>>>> http://www.infinispan.org
>>>>> http://www.jbosscache.org
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> infinispan-dev mailing list
>>>>> infinispan-dev at lists.jboss.org
>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>> --
>>>> Galder Zamarreño
>>>> Sr. Software Engineer
>>>> Infinispan, JBoss Cache
>>>>
>>
>

-- 
Galder Zamarreño
Sr. Software Engineer
Infinispan, JBoss Cache