[infinispan-dev] Infinispan and change data capture

[Emmanuel Bernard]

> On 16 Dec 2016, at 16:25, Emmanuel Bernard <emmanuel at hibernate.org> wrote:
> 
> 
>> On 16 Dec 2016, at 13:38, Tristan Tarrant <ttarrant at redhat.com> wrote:
>> 
>> On 16/12/16 13:12, Emmanuel Bernard wrote:
>>> 
>>>> On 16 Dec 2016, at 09:48, Tristan Tarrant <ttarrant at redhat.com> wrote:
>>>> 
>>>> On 16/12/16 09:34, Emmanuel Bernard wrote:
>>>>>> Yes, the above design is what sprung to mind initially. Not sure about
>>>>>> the need of keeping the log in memory, as we would probably need some
>>>>>> form of persistent log for cache shutdown. Since this looks a lot like
>>>>>> the append-log of the Artemis journal, maybe we could use that.
>>>>> 
>>>>> Well, when the cache is shut down, don’t we have time to empty the in-memory log?
>>>> 
>>>> Cache shutdown should not be deferred because there is a backlog of
>>>> events that haven't been forwarded to Debezium, so we would want to pick
>>>> up from where we were when we restart the cache.
>>> 
>>> But you’re willing to wait for the Artemis journal finish writing? I don’t quite see the difference.
>> 
>> I'm thinking about the case where Debezium is temporarily not able to 
>> collect the changes.
> 
> I’m thinking about the case where Artemis is not able to collect the changes. How is that different? :)

So to answer my own questions.

The Emmanuel Dec 9th proposal handles I think the case of topology changes and nodes going down.
For a cluster-wide shutdown with no time to flush queues, I think both Artemis and the local debezium talking to remote Kafka will be in trouble.

The main difference is I imagine that your Artemis log will be local to the node being shut down. But that would mean a complex system to restart with the unflushed queue from all node that were shut down. Are we there yet?