Re: [infinispan-dev] Infinispan and change data capture

On 16 Dec 2016, at 08:46, Tristan Tarrant <ttarrant(a)redhat.com&gt; wrote: On 09/12/16 18:25, Emmanuel Bernard wrote: > The total order would not be global but per key. > Each node has a Debezium connector instance embedded that listens to the > operations happening (primary and replicas alike). > All of this process is happening async compared to the operation. > Per key, a log of operations is kept in memory (it contains the key, the > operation, the operation unique id and a ack status. > If on the key owner, the operation is written by the Debezium connector > to Kafka when it has been acked (whatever that means is where I'm less > knowledgable - too many bi-cache, tri-cache and quadri latency mixed in > my brain). > On a replica, the kafka partition is read regularly to clear the > in-memory log from operations stored in Kafka > If the replica becomes the owner, it reads the kafka partition to see Yes, the above design is what sprung to mind initially. Not sure about the need of keeping the log in memory, as we would probably need some form of persistent log for cache shutdown. Since this looks a lot like the append-log of the Artemis journal, maybe we could use that.

One thing I missed, is whether Debezium is interested in transaction boundaries.