[infinispan-dev] Infinispan and change data capture

[Emmanuel Bernard]

> On 15 Dec 2016, at 16:09, Sanne Grinovero <sanne at infinispan.org> wrote:
> 
> Thanks Randall,
> those clarifications have been great.
> 
> Emmanuel: some of your statements conflict with Randall's
> clarifications and with the feasibility points I've been pointing at.
> You say "collect *all* changes". I've been questioning that Infinispan
> can not keep *all* changes around for a given single key;

Yes I want a stronger contract than what Randall said we could do at minimum.
But again, it’s not Infinispan keeping all changes around. It Infinispan keeping changes around long enough for an external system to collect them. My email from Dec 9th offers a back-pressure mechanism.
I rephrased it in my reply to Gustavo today, so hopefully that clears things up.

> I understand
> we'd allow clients to retrieve streams of changes persisted into
> Kafka, but we need to be clear that we won't be handling *all* changes
> to Kafka (nor to Debezium),

Why not all changes again? Remember Debezium will be embedded so it will only miss a change iif the node crashes or does not see the change itself. And that’s where the temp queue and back-pressure system on the replicas come into play.

> so the magic these can do is somewhat
> limited. They can certainly expand on the capabilities that Infinispan
> would provide on its own, but some of the use cases which Gustavo
> mentioned would not be suitable.

OK, I’m catching up and I’ve just read Gustavo’s proposal of https://github.com/infinispan/infinispan/wiki/Remote-Listeners-improvement-proposal and I understand better some of the comments in the thread.

So the Emmanuel, Dec 9th proposal does not address all of the use cases Gustavo had in mind. I’m neutral to the native event store idea, I had assumed that was a no-go from a team choice PoV.
Gustavo describes the necessary back-pressure mechanism that all event store consumers would have to use. It feels like a longer shot and I’m concerned about the ability to cap the event store size. In the Emmanuel, Dec 9th proposal, at least the caping is addressed and it is expected that the in-memory structure will remain small. If we’re aiming at use cases with cap-less stores, then I think Kafka is a better long term storage than what Infinispan could offer.

Emmanuel