[infinispan-dev] Infinispan and change data capture

[Tristan Tarrant]

On 09/12/16 18:25, Emmanuel Bernard wrote:
> The total order would not be global but per key.
> Each node has a Debezium connector instance embedded that listens to the
> operations happening (primary and replicas alike).
> All of this process is happening async compared to the operation.
> Per key, a log of operations is kept in memory (it contains the key, the
> operation, the operation unique id and a ack status.
> If on the key owner, the operation is written by the Debezium connector
> to Kafka when it has been acked (whatever that means is where I'm less
> knowledgable - too many bi-cache, tri-cache and quadri latency mixed in
> my brain).
> On a replica, the kafka partition is read regularly to clear the
> in-memory log from operations stored in Kafka
> If the replica becomes the owner, it reads the kafka partition to see

Yes, the above design is what sprung to mind initially. Not sure about 
the need of keeping the log in memory, as we would probably need some 
form of persistent log for cache shutdown. Since this looks a lot like 
the append-log of the Artemis journal, maybe we could use that.
One thing I missed, is whether Debezium is interested in transaction 
boundaries.

Tristan

-- 
Tristan Tarrant
Infinispan Lead
JBoss, a division of Red Hat