<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Dec 15, 2016 at 5:47 PM, Emmanuel Bernard <span dir="ltr"><<a href="mailto:emmanuel@hibernate.org" target="_blank">emmanuel@hibernate.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><span class="gmail-"><br>
> On 15 Dec 2016, at 16:09, Sanne Grinovero <<a href="mailto:sanne@infinispan.org">sanne@infinispan.org</a>> wrote:<br>
><br>
> Thanks Randall,<br>
> those clarifications have been great.<br>
><br>
> Emmanuel: some of your statements conflict with Randall's<br>
> clarifications and with the feasibility points I've been pointing at.<br>
> You say "collect *all* changes". I've been questioning that Infinispan<br>
> can not keep *all* changes around for a given single key;<br>
<br>
</span>Yes I want a stronger contract than what Randall said we could do at minimum.<br>
But again, it’s not Infinispan keeping all changes around. It Infinispan keeping changes around long enough for an external system to collect them. My email from Dec 9th offers a back-pressure mechanism.<br>
I rephrased it in my reply to Gustavo today, so hopefully that clears things up.<br>
<span class="gmail-"><br>
> I understand<br>
> we'd allow clients to retrieve streams of changes persisted into<br>
> Kafka, but we need to be clear that we won't be handling *all* changes<br>
> to Kafka (nor to Debezium),<br>
<br>
</span>Why not all changes again? Remember Debezium will be embedded so it will only miss a change iif the node crashes or does not see the change itself. And that’s where the temp queue and back-pressure system on the replicas come into play.<br>
<span class="gmail-"><br>
> so the magic these can do is somewhat<br>
> limited. They can certainly expand on the capabilities that Infinispan<br>
> would provide on its own, but some of the use cases which Gustavo<br>
> mentioned would not be suitable.<br>
<br>
</span>OK, I’m catching up and I’ve just read Gustavo’s proposal of <a href="https://github.com/infinispan/infinispan/wiki/Remote-Listeners-improvement-proposal" rel="noreferrer" target="_blank">https://github.com/infinispan/<wbr>infinispan/wiki/Remote-<wbr>Listeners-improvement-proposal</a> and I understand better some of the comments in the thread.<br>
<br>
So the Emmanuel, Dec 9th proposal does not address all of the use cases Gustavo had in mind. I’m neutral to the native event store idea, I had assumed that was a no-go from a team choice PoV.<br>
Gustavo describes the necessary back-pressure mechanism that all event store consumers would have to use. It feels like a longer shot and I’m concerned about the ability to cap the event store size. In the Emmanuel, Dec 9th proposal, at least the caping is addressed and it is expected that the in-memory structure will remain small. If we’re aiming at use cases with cap-less stores, then I think Kafka is a better long term storage than what Infinispan could offer.<br></blockquote><div><br></div><div>I mentioned "off-heap *bounded* event storage" in the proposal. Whether this is backed by a cache or a Lucene index, up for discussion.<br><br>The idea is basically what you are describing: the native event storage time span is configurable. Even 30min worth of events <br>would improve things: a client that disconnects for 2 seconds and comes back later would not need to either 1) loose events <br>that happened in between or 2) get the whole data of the cache again.<br><br></div><div>For a system that needs *all* the events from Infinispan since the dawn of time, the recommendation would be to plug something like Debezium<br></div><div>that consumes ISPN data regularly (and retry when disconnected), and save the events to a more suitable long-term data backend, a time series database.<br><br><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<span class="gmail-HOEnZb"><font color="#888888"><br>
Emmanuel<br>
</font></span><div class="gmail-HOEnZb"><div class="gmail-h5"><br>
<br>
______________________________<wbr>_________________<br>
infinispan-dev mailing list<br>
<a href="mailto:infinispan-dev@lists.jboss.org">infinispan-dev@lists.jboss.org</a><br>
<a href="https://lists.jboss.org/mailman/listinfo/infinispan-dev" rel="noreferrer" target="_blank">https://lists.jboss.org/<wbr>mailman/listinfo/infinispan-<wbr>dev</a></div></div></blockquote></div><br></div></div>