[infinispan-dev] Distributed Counter Discussion

[Bela Ban]

What are the requirements? To offer a cluster wide counter from an 
Infinispan cache? Or is this going to be used internally, by Infinispan?

I've implemented 2 counters, in JGroups and in jgroups-raft. Here are 
their properties:

#1 JGroups counter:
- Cluster wide _named_ counters with atomicity guarantees
- A counter is always managed by the current coordinator
- In a cluster split, you can end up with different counter instances 
(one per coordinator)
- When a cluster partition heals, the current coordinator's counter 
'wins', ie. values from the other counters are lost, even if they were 
updated last

#2 jgroups-raft counter:
- Counter is managed by the leader
- A counter will not be accessible when there's no leader (no majority): 
this is CP in CAP
- Counters cannot diverge in split brain scenarios; there's always at 
most 1 counter (or 0)

If you can live with multiple counters being present in light of network 
partitiond, then pick #1 as this is faster than #2 (which requires disk 
writes for updates), and is always available, whereas #2 can be 
non-available.

I'm thinking about changing the way #1 works. Currently, it uses the 
next-in-line to backup a counter value on each update. I'd like to 
remove this and replace it with a reconciliation round when the 
coordinator changes: the new coordinator asks all members for their 
current counter values. Requires a bit more work on a view change, but 
less in steady state mode. Perhaps I can make it configurable.


On 14/03/16 20:14, Pedro Ruivo wrote:
> Hi everybody,
>
> Discussion about distributed counters.
>
> == Public API ==
>
> interface Counter
>
> String getName() //counter name
> long get()	 //current value. may return stale value due to concurrent
> operations to other nodes.
> void increment() //async or sync increment. default add(1)
> void decrement() //async or sync decrement. default add(-1)
> void add(long)   //async or sync add.
> void reset()     //resets to initial value
>
> Note: Tried to make the interface as simple as possible with support for
> sync and async operations. To avoid any confusion, I consider an async
> operation as happening somewhat in the future, i.e. eventually
> increments/decrements.
> The sync operation happens somewhat during the method execution.
>
> interface AtomiCounter extends Counter
>
> long addAndGet()       //adds a returns the new value. sync operation
> long incrementAndGet() //increments and returns the new value. sync
> operation. default addAndGet(1)
> long decrementAndGet() //decrements and returns the new value. sync
> operation. default addAndGet(-1)
>
> interface AdvancedCounter extends Counter
>
> long getMin/MaxThreshold() //returns the min and max threshold value
> void add/removeListener()  //adds a listener that is invoked when the
> value change. Can be extended to notify when it is "reseted" and when
> the threshold is reached.
>
> Note: should this interface be splitted?
>
> == Details ==
>
> This is what I have in mind. Two counter managers: one based on JGroups
> counter and another one based on Infinispan cache.
> The first one creates AtomicCounters and it first perfectly. All
> counters are created with an initial value (zero by default)
> The second generates counters with all the options available. It can mix
> sync/async operation and all counters will be in the same cache. The
> cache will be configure by us and it would be an internal cache. This
> will use all the features available in the cache.
>
> Configuration-wise, I'm thinking about 2 parameters: number of backups
> and timeout (for sync operations).
>
> So, comment bellow and let me know alternatives, improvement or if I
> missed something.
>
> ps. I also consider implement a counter based on JGroups-raft but I
> believe it is an overkill.
> ps2. sorry for the long email :( I tried to be shorter as possible.
>
> Cheers,
> Pedro
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>

-- 
Bela Ban, JGroups lead (http://www.jgroups.org)