Re: [infinispan-dev] Infinispan and change data capture
On 15 Dec 2016, at 16:09, Sanne Grinovero
<sanne(a)infinispan.org> wrote:
Thanks Randall,
those clarifications have been great.
Emmanuel: some of your statements conflict with Randall's
clarifications and with the feasibility points I've been pointing at.
You say "collect *all* changes". I've been questioning that Infinispan
can not keep *all* changes around for a given single key;
Yes I want a stronger contract than what Randall said we could do at minimum.
But again, it’s not Infinispan keeping all changes around. It Infinispan keeping changes
around long enough for an external system to collect them. My email from Dec 9th offers a
back-pressure mechanism.
I rephrased it in my reply to Gustavo today, so hopefully that clears things up.
I understand
we'd allow clients to retrieve streams of changes persisted into
Kafka, but we need to be clear that we won't be handling *all* changes
to Kafka (nor to Debezium),
Why not all changes again? Remember Debezium will be embedded so it will only miss a
change iif the node crashes or does not see the change itself. And that’s where the temp
queue and back-pressure system on the replicas come into play.
so the magic these can do is somewhat
limited. They can certainly expand on the capabilities that Infinispan
would provide on its own, but some of the use cases which Gustavo
mentioned would not be suitable.
OK, I’m catching up and I’ve just read Gustavo’s proposal of
https://github.com/infinispan/infinispan/wiki/Remote-Listeners-improvemen... and I
understand better some of the comments in the thread.
So the Emmanuel, Dec 9th proposal does not address all of the use cases Gustavo had in
mind. I’m neutral to the native event store idea, I had assumed that was a no-go from a
team choice PoV.
Gustavo describes the necessary back-pressure mechanism that all event store consumers
would have to use. It feels like a longer shot and I’m concerned about the ability to cap
the event store size. In the Emmanuel, Dec 9th proposal, at least the caping is addressed
and it is expected that the in-memory structure will remain small. If we’re aiming at use
cases with cap-less stores, then I think Kafka is a better long term storage than what
Infinispan could offer.
Emmanuel