[infinispan-dev] Distributed Counter Discussion

Tue Mar 15 10:15:22 EDT 2016

> On Mar 15, 2016, at 3:26 AM, Radim Vansa <rvansa at redhat.com> wrote:
> 
> I second Sanne's opinion about the sync/async API: make the API express 
> the synchronicity directly. I would even propose that for synchronous 
> methods, there's only the CompletableFuture<Void> or 
> CompletableFuture<Long> variant; we are already reaching the concurrency 
> limits of application that lets threads block, so let's give user a hint 
> that he should use such notification API. If anyone prefers the sync 
> variant, he can always use get().
> 
> Let's settle on some nomenclature, too, because these JGroups-counters 
> and RAFT-counters don't have commonly known properties. It almost seems 
> that the term *counter* is so overloaded that we shouldn't use it all.
> 
> There is a widely known j.u.c.AtomicLong, so if we want to implement 
> that (strict JMM-like properties), let's call it AtomicLong. Does not 
> have to follow j.u.c.AtomicLong API, but it's definitely CP.
> 
> Something providing unique values should be called Sequence. This will 
> be probably the one batching ranges (therefore possibly with gaps). By 
> default non-monotonic, but monotonicity could be a ctor arg (in a 
> similar way as fairness is set for j.u.concurrent.* classes).

+1 for differentiating in the API between a counter that a client increments (or resets) and a sequence generator that a client can use to obtain (non)monotonic values.

> 
> As for CRDTs, I can't imagine how this could be easily built on top of 
> current 'passive' cache (without any syncing). But as for names *CRDT 
> counter* is explanatory enough.
> 
> If someone needs quotas, let's create Quota according to their needs 
> (soft and hard threshold, fluent operation below soft, some jitter when 
> above that). It seems that this will be closest to Sequence by reserving 
> some ranges. Don't let them shoot themselves into foot with some liger 
> counter.

+1 for creating a specific interface to enable quotas. Perhaps it is implemented with some more general purpose functionality that might be exposed in the future, but for now why not keep it narrowly focused.

> 
> And I hope you'll build these on top of the functional API, with at most 
> one RPC per operation.
> 
> Radim
> 
> On 03/14/2016 10:27 PM, Sanne Grinovero wrote:
>> Great starting point!
>> Some comments inline:
>> 
>> On 14 March 2016 at 19:14, Pedro Ruivo <pedro at infinispan.org> wrote:
>>> Hi everybody,
>>> 
>>> Discussion about distributed counters.
>>> 
>>> == Public API ==
>>> 
>>> interface Counter
>> As a user, how do I get a Counter instance? From a the CacheContainer interface?
>> 
>> Will they have their own configuration section in the configuration file?
>> 
>>> String getName() //counter name
>>> long get()       //current value. may return stale value due to concurrent
>>> operations to other nodes.
>> This is what puzzles me the most. I'm not sure if the feature is
>> actually useful, unless we can clearly state how far outdated the
>> value could be.
>> 
>> I think a slightly more formal definition would be in order. For
>> example I think it would be acceptable to say that this will return a
>> value from the range of values the primary owner of this counter was
>> holding in the timeframe between the method is being invoked and the
>> time the value is returned.
>> 
>> Could it optionally be integrated with Total Order ? Transactions?
>> 
>>> void increment() //async or sync increment. default add(1)
>>> void decrement() //async or sync decrement. default add(-1)
>>> void add(long)   //async or sync add.
>>> void reset()     //resets to initial value
>>> 
>>> Note: Tried to make the interface as simple as possible with support for
>>> sync and async operations. To avoid any confusion, I consider an async
>>> operation as happening somewhat in the future, i.e. eventually
>>> increments/decrements.
>>> The sync operation happens somewhat during the method execution.
>>> 
>>> interface AtomiCounter extends Counter
>>> 
>>> long addAndGet()       //adds a returns the new value. sync operation
>>> long incrementAndGet() //increments and returns the new value. sync
>>> operation. default addAndGet(1)
>>> long decrementAndGet() //decrements and returns the new value. sync
>>> operation. default addAndGet(-1)
>>> 
>>> interface AdvancedCounter extends Counter
>>> 
>>> long getMin/MaxThreshold() //returns the min and max threshold value
>> "threshold" ??
>> 
>>> void add/removeListener()  //adds a listener that is invoked when the
>>> value change. Can be extended to notify when it is "reseted" and when
>>> the threshold is reached.
>>> 
>>> Note: should this interface be splitted?
>> I'd prefer a single interface, with reduced redundancy.
>> For example, is there really a benefit in having a "void increment()"
>> and also a "long addAndGet()" ? [Besides The fact that only the first
>> one can benefit of an async option]
>> 
>> Besides, I am no longer sure that it's good thing that methods in
>> Infinispan can be async vs sync depending on configuration switches;
>> I'd rather make it explicit in the signature and simplify the
>> configuration by removing such a flag.
>> 
>> Making the methods which are async-capable to look "explicitly async"
>> should also allow us to add completeable futures & similar.
>> 
>>> == Details ==
>>> 
>>> This is what I have in mind. Two counter managers: one based on JGroups
>>> counter and another one based on Infinispan cache.
>>> The first one creates AtomicCounters and it first perfectly. All
>>> counters are created with an initial value (zero by default)
>>> The second generates counters with all the options available. It can mix
>>> sync/async operation and all counters will be in the same cache. The
>>> cache will be configure by us and it would be an internal cache. This
>>> will use all the features available in the cache.
>> Rather than a split organized on the backing implementation details,
>> I'd prefer to see a split based on purpose of the counter.
>> 
>> For example JGroups has different styles of counters: one might need
>> one to do an atomic increment which is monotonically increasing across
>> the whole cluster - from the point of view from an omniscent observer
>> - but in many cases we just need the generation of a guarantee of a
>> monotonic unique number: that allows for example to have each node
>> pre-allocate a range of values and pre-fetch a new block while its
>> running out of values.
>> 
>> For some systems this is not acceptable because a failing server might
>> result in some of its allocated numbers to not ever be used, so
>> creating gaps; for others it's not acceptable that the figures
>> returned by the "monotonic counter" are not monotonic across each
>> other when confronting results from different cluster nodes, but for
>> many use cases that's acceptable. My point being that we need to be
>> able to choose between these possible semantics.
>> 
>> As an Infinispan consumer being able to express such details is
>> essential, while having to choose between "backed by JGroups counters"
>> or not is irrelevant and actually makes it harder.
>> 
>> In terms of Infinispan integration, the most important point I'd like
>> to see is persistence: i.e. make sure you can store them in a
>> CacheStore.
>> How we do that efficiently will also affect design; for example I was
>> expecting to see a Counter as something which is stored in a Cache, so
>> inheriting details such as configured CacheStore(s) and number of
>> owners.
>> 
>> Thanks for starting this!
>> 
>> Sanne
>> 
>>> Configuration-wise, I'm thinking about 2 parameters: number of backups
>>> and timeout (for sync operations).
>>> 
>>> So, comment bellow and let me know alternatives, improvement or if I
>>> missed something.
>>> 
>>> ps. I also consider implement a counter based on JGroups-raft but I
>>> believe it is an overkill.
>>> ps2. sorry for the long email :( I tried to be shorter as possible.
>>> 
>>> Cheers,
>>> Pedro
>>> _______________________________________________
>>> infinispan-dev mailing list
>>> infinispan-dev at lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> 
> 
> -- 
> Radim Vansa <rvansa at redhat.com>
> JBoss Performance Team
> 
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev