What are the requirements? What are the distributed counters for? Is the counter to be monotonically increasing? Can there be any missed values? Does the counter need to increment and decrement? What is the *smallest* API you need initially?

There are two choices when implementing a distributed counter: use central coordination (like JGroups counters), or use independent counters on separate machines that will eventually converge to the correct value (CRDTs). Coordinated counters are expensive and therefore slow, and can suffer from problems during network or cluster problems. For example, what happens during a split brain? OTOH, CRDTs are decentralized so therefore are very fast, easily merged, and fault tolerant; they’re excellent when counting things that are occurring independently and therefore may be more suited for monitoring/metrics/accumulators/etc.  Both have very different behaviors under ideal and failure scenarios, have different performance and consistency guarantees, and are useful in different scenarios. Make sure you choose accordingly.

For information about CRDTs, make sure you’ve read the CRDT paper by Shapiro: http://hal.upmc.fr/inria-00555588/document 

Randall

On Mar 14, 2016, at 2:14 PM, Pedro Ruivo <pedro@infinispan.org> wrote:

Hi everybody,

Discussion about distributed counters.

== Public API ==

interface Counter

String getName() //counter name
long get() //current value. may return stale value due to concurrent
operations to other nodes.
void increment() //async or sync increment. default add(1)
void decrement() //async or sync decrement. default add(-1)
void add(long)   //async or sync add.
void reset()     //resets to initial value

Note: Tried to make the interface as simple as possible with support for
sync and async operations. To avoid any confusion, I consider an async
operation as happening somewhat in the future, i.e. eventually
increments/decrements.
The sync operation happens somewhat during the method execution.

interface AtomiCounter extends Counter

long addAndGet()       //adds a returns the new value. sync operation
long incrementAndGet() //increments and returns the new value. sync
operation. default addAndGet(1)
long decrementAndGet() //decrements and returns the new value. sync
operation. default addAndGet(-1)

interface AdvancedCounter extends Counter

long getMin/MaxThreshold() //returns the min and max threshold value
void add/removeListener()  //adds a listener that is invoked when the
value change. Can be extended to notify when it is "reseted" and when
the threshold is reached.

Note: should this interface be splitted?

== Details ==

This is what I have in mind. Two counter managers: one based on JGroups
counter and another one based on Infinispan cache.
The first one creates AtomicCounters and it first perfectly. All
counters are created with an initial value (zero by default)
The second generates counters with all the options available. It can mix
sync/async operation and all counters will be in the same cache. The
cache will be configure by us and it would be an internal cache. This
will use all the features available in the cache.

Configuration-wise, I'm thinking about 2 parameters: number of backups
and timeout (for sync operations).

So, comment bellow and let me know alternatives, improvement or if I
missed something.

ps. I also consider implement a counter based on JGroups-raft but I
believe it is an overkill.
ps2. sorry for the long email :( I tried to be shorter as possible.

Cheers,
Pedro
_______________________________________________
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev