[infinispan-dev] Infinispan and change data capture
Tristan Tarrant
ttarrant at redhat.com
Fri Dec 16 02:46:45 EST 2016
On 09/12/16 18:25, Emmanuel Bernard wrote:
> The total order would not be global but per key.
> Each node has a Debezium connector instance embedded that listens to the
> operations happening (primary and replicas alike).
> All of this process is happening async compared to the operation.
> Per key, a log of operations is kept in memory (it contains the key, the
> operation, the operation unique id and a ack status.
> If on the key owner, the operation is written by the Debezium connector
> to Kafka when it has been acked (whatever that means is where I'm less
> knowledgable - too many bi-cache, tri-cache and quadri latency mixed in
> my brain).
> On a replica, the kafka partition is read regularly to clear the
> in-memory log from operations stored in Kafka
> If the replica becomes the owner, it reads the kafka partition to see
Yes, the above design is what sprung to mind initially. Not sure about
the need of keeping the log in memory, as we would probably need some
form of persistent log for cache shutdown. Since this looks a lot like
the append-log of the Artemis journal, maybe we could use that.
One thing I missed, is whether Debezium is interested in transaction
boundaries.
Tristan
--
Tristan Tarrant
Infinispan Lead
JBoss, a division of Red Hat
More information about the infinispan-dev
mailing list