[infinispan-dev] Infinispan and change data capture

[Gustavo Fernandes]

On Thu, Dec 15, 2016 at 5:47 PM, Emmanuel Bernard <emmanuel at hibernate.org>
wrote:

>
> > On 15 Dec 2016, at 16:09, Sanne Grinovero <sanne at infinispan.org> wrote:
> >
> > Thanks Randall,
> > those clarifications have been great.
> >
> > Emmanuel: some of your statements conflict with Randall's
> > clarifications and with the feasibility points I've been pointing at.
> > You say "collect *all* changes". I've been questioning that Infinispan
> > can not keep *all* changes around for a given single key;
>
> Yes I want a stronger contract than what Randall said we could do at
> minimum.
> But again, it’s not Infinispan keeping all changes around. It Infinispan
> keeping changes around long enough for an external system to collect them.
> My email from Dec 9th offers a back-pressure mechanism.
> I rephrased it in my reply to Gustavo today, so hopefully that clears
> things up.
>
> > I understand
> > we'd allow clients to retrieve streams of changes persisted into
> > Kafka, but we need to be clear that we won't be handling *all* changes
> > to Kafka (nor to Debezium),
>
> Why not all changes again? Remember Debezium will be embedded so it will
> only miss a change iif the node crashes or does not see the change itself.
> And that’s where the temp queue and back-pressure system on the replicas
> come into play.
>
> > so the magic these can do is somewhat
> > limited. They can certainly expand on the capabilities that Infinispan
> > would provide on its own, but some of the use cases which Gustavo
> > mentioned would not be suitable.
>
> OK, I’m catching up and I’ve just read Gustavo’s proposal of
> https://github.com/infinispan/infinispan/wiki/Remote-
> Listeners-improvement-proposal and I understand better some of the
> comments in the thread.
>
> So the Emmanuel, Dec 9th proposal does not address all of the use cases
> Gustavo had in mind. I’m neutral to the native event store idea, I had
> assumed that was a no-go from a team choice PoV.
> Gustavo describes the necessary back-pressure mechanism that all event
> store consumers would have to use. It feels like a longer shot and I’m
> concerned about the ability to cap the event store size. In the Emmanuel,
> Dec 9th proposal, at least the caping is addressed and it is expected that
> the in-memory structure will remain small. If we’re aiming at use cases
> with cap-less stores, then I think Kafka is a better long term storage than
> what Infinispan could offer.
>

I mentioned "off-heap *bounded* event storage" in the proposal. Whether
this is backed by a cache or a Lucene index, up for discussion.

The idea is basically what you are describing: the native event storage
time span is configurable.  Even 30min worth of events
would improve things: a client that disconnects for 2 seconds and comes
back later would not need to either 1) loose events
that happened in between or 2) get the whole data of the cache again.

For a system that needs *all* the events from Infinispan since the dawn of
time, the recommendation would be to plug something like Debezium
that consumes ISPN data regularly (and retry when disconnected), and save
the events to a more suitable long-term data backend, a time series
database.



> Emmanuel
>
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/infinispan-dev/attachments/20161215/cbbdac6d/attachment-0001.html