[infinispan-dev] Infinispan and change data capture

[Emmanuel Bernard]

> On 16 Dec 2016, at 08:46, Tristan Tarrant <ttarrant at redhat.com> wrote:
> 
> On 09/12/16 18:25, Emmanuel Bernard wrote:
>> The total order would not be global but per key.
>> Each node has a Debezium connector instance embedded that listens to the
>> operations happening (primary and replicas alike).
>> All of this process is happening async compared to the operation.
>> Per key, a log of operations is kept in memory (it contains the key, the
>> operation, the operation unique id and a ack status.
>> If on the key owner, the operation is written by the Debezium connector
>> to Kafka when it has been acked (whatever that means is where I'm less
>> knowledgable - too many bi-cache, tri-cache and quadri latency mixed in
>> my brain).
>> On a replica, the kafka partition is read regularly to clear the
>> in-memory log from operations stored in Kafka
>> If the replica becomes the owner, it reads the kafka partition to see
> 
> Yes, the above design is what sprung to mind initially. Not sure about 
> the need of keeping the log in memory, as we would probably need some 
> form of persistent log for cache shutdown. Since this looks a lot like 
> the append-log of the Artemis journal, maybe we could use that.

Well, when the cache is shut down, don’t we have time to empty the in-memory log?

> One thing I missed, is whether Debezium is interested in transaction 
> boundaries.


It’s an information we capture in the RDBMS case. So yes Debezium would be interested if available.