Re: [infinispan-dev] Infinispan and change data capture
On 16 Dec 2016, at 16:25, Emmanuel Bernard
<emmanuel(a)hibernate.org> wrote:
> On 16 Dec 2016, at 13:38, Tristan Tarrant <ttarrant(a)redhat.com> wrote:
>
> On 16/12/16 13:12, Emmanuel Bernard wrote:
>>
>>> On 16 Dec 2016, at 09:48, Tristan Tarrant <ttarrant(a)redhat.com> wrote:
>>>
>>> On 16/12/16 09:34, Emmanuel Bernard wrote:
>>>>> Yes, the above design is what sprung to mind initially. Not sure
about
>>>>> the need of keeping the log in memory, as we would probably need
some
>>>>> form of persistent log for cache shutdown. Since this looks a lot
like
>>>>> the append-log of the Artemis journal, maybe we could use that.
>>>>
>>>> Well, when the cache is shut down, don’t we have time to empty the
in-memory log?
>>>
>>> Cache shutdown should not be deferred because there is a backlog of
>>> events that haven't been forwarded to Debezium, so we would want to pick
>>> up from where we were when we restart the cache.
>>
>> But you’re willing to wait for the Artemis journal finish writing? I don’t quite
see the difference.
>
> I'm thinking about the case where Debezium is temporarily not able to
> collect the changes.
I’m thinking about the case where Artemis is not able to collect the changes. How is that
different? :)
So to answer my own questions.
The Emmanuel Dec 9th proposal handles I think the case of topology changes and nodes going
down.
For a cluster-wide shutdown with no time to flush queues, I think both Artemis and the
local debezium talking to remote Kafka will be in trouble.
The main difference is I imagine that your Artemis log will be local to the node being
shut down. But that would mean a complex system to restart with the unflushed queue from
all node that were shut down. Are we there yet?