<div dir="ltr">On Thu, Dec 15, 2016 at 9:54 AM, Emmanuel Bernard <span dir="ltr"><<a href="mailto:emmanuel@hibernate.org" target="_blank">emmanuel@hibernate.org</a>></span> wrote:<br><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">The goal is as followed: allow to collect all changes to push them to Debezium and thus Kafka.<br>
<br>
This need does not require to remember all changes since the beginning of time in Infinispan. Just enough to:<br>
- let Kafka catchup assuming it is the bottleneck<br>
- let us not lose a change in Kafka when it happened in Infinispan (coordinator, owner, replicas dying)<br>
<br>
The ability to read back history would then be handled by the Debezium / Kafka tail, not infinispan itself.<br>
<br></blockquote><div><br></div><div>Having an embedded Debezium connector pushing everything to Kafka sounds cool, but what impact would it bring to the other stream consumers:<br><br>* Remote listeners, which is supported in several clients apart from Java<br></div><div>* Continuous Queries (the same)<br></div><div>* Spark Stream<br></div><div>* Other eventual 3rd party stream processors: Apache Flick, Storm, etc.<br><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Check my email on this tread from Dec 9th.<br>
<div class="gmail-HOEnZb"><div class="gmail-h5"><br>
> On 12 Dec 2016, at 16:13, Sanne Grinovero <<a href="mailto:sanne@infinispan.org">sanne@infinispan.org</a>> wrote:<br>
><br>
> I'm reading many clever suggestions for various aspects of such a<br>
> system, but I fail to see a clear definition of the goal.<br>
><br>
>> From Randall's opening email I understand how MySQL does this, but<br>
> it's an example and I'm not sure which aspects are implementation<br>
> details of how MySQL happens to accomplish this, and which aspects are<br>
> requirements for the Infinispan enhancement proposals.<br>
><br>
> I remember a meeting with Manik Surtani, Jonathan Halliday and Mark<br>
> Little, whose outcome was a general agreement that Infinispan would<br>
> eventually need both tombstones and versioned entries, not just for<br>
> change data capture but to improve several other aspects;<br>
> unfortunately that was in December 2010 and never became a priority,<br>
> but the benefits are clear.<br>
> The complexities which have put off such plans lie in the "garbage<br>
> collection", aka the need to not grow the history without bounds, and<br>
> have to drop or compact history.<br>
><br>
> So I'm definitely sold on the need to add a certain amount of history,<br>
> but we need to define how much of this history is expected to be held.<br>
><br>
> In short, what's the ultimate goal? I see two main but different<br>
> options intertwined:<br>
> - allow to synchronize the *final state* of a replica<br>
> - inspect specific changes<br>
><br>
> For the first case, it would be enough for us to be able to provide a<br>
> "squashed history" (as in Git squash), but we'd need to keep versioned<br>
> shapshots around and someone needs to tell you which ones can be<br>
> garbage collected.<br>
> For example when a key is: written, updated, updated, deleted since<br>
> the snapshot, we'll send only "deleted" as the intermediary states are<br>
> irrelevant.<br>
> For the second case, say the goal is to inspect fluctuations of price<br>
> variations of some item, then the intermediary states are not<br>
> irrelevant.<br>
><br>
> Which one will we want to solve? Both?<br>
> Personally the attempt of solving the second one seems like a huge<br>
> pivot of the project, the current data-structures and storage are not<br>
> designed for this. I see the value of such benefits, but maybe<br>
> Infinispan is not the right tool for such a problem.<br>
><br>
> I'd prefer to focus on the benefits of the squashed history, and have<br>
> versioned entries soon, but even in that case we need to define which<br>
> versions need to be kept around, and how garbage collection /<br>
> vacuuming is handled.<br>
> This can be designed to be transparent to the client: handled as an<br>
> internal implementation detail which we use to improve performance of<br>
> Infinispan itself, or it can be exposed to clients to implement change<br>
> data capture, but in this case we need to track which clients are<br>
> still going to need an older snapshot; this has an impact as clients<br>
> would need to be registered, and has a significant impact on the<br>
> storage strategies.<br>
><br>
> Within Kafka the log compaction strategies are configurable; I have no<br>
> experience with Kafka but the docs seem to suggest that it's most<br>
> often used to provide the last known value of each key. That would be<br>
> doable for us, but Kafka also does allow optionally for wider scope<br>
> retention strategies: can we agree that that would not be an option<br>
> with Infinispan? If not, these goals need to be clarified.<br>
><br>
> My main concern is that if we don't limit the scope of which<br>
> capabilities we want Infinispan to provide, it risks to become the<br>
> same thing as Kafka, rather than integrating with it. I don't think we<br>
> want to pivot all our storage design into efficiently treating large<br>
> scale logs.<br>
><br>
> In short, I'd like to see an agreement that analyzing e.g.<br>
> fluctuations in stock prices would be a non-goal, if these are stored<br>
> as {"stock name", value} key/value pairs. One could still implement<br>
> such a thing by using a more sophisticated model, just don't expect to<br>
> be able to see all intermediary values each entry has ever had since<br>
> the key was first used.<br>
><br>
> # Commenting on specific proposals<br>
><br>
> On ID generation: I'd definitely go with IDs per segment rather than<br>
> IDs per key for the purpose of change data capture. If you go with<br>
> independent IDs per key, the client would need to keep track of each<br>
> version of each entry, which has an high overhead and degree of<br>
> complexity for the clients.<br>
> On the other hand, we already guarantee that each segment is managed<br>
> by a single primary owner, so attaching the "segment transaction id"<br>
> to each internal entry being changed can be implemented efficiently by<br>
> Infinispan.<br>
> Segment ownership handoff needs to be highly consistent during cluster<br>
> topology changes, but that requirement already exists; we'd just need<br>
> to make sure that this monotonic counter is included during the<br>
> handoff of the responsibility as primary owner of a segment.<br>
><br>
> Thanks,<br>
><br>
> Sanne<br>
><br>
><br>
><br>
><br>
><br>
> On 12 December 2016 at 13:58, Gustavo Fernandes <<a href="mailto:gustavo@infinispan.org">gustavo@infinispan.org</a>> wrote:<br>
>><br>
>><br>
>> On Fri, Dec 9, 2016 at 9:13 AM, Radim Vansa <<a href="mailto:rvansa@redhat.com">rvansa@redhat.com</a>> wrote:<br>
>>><br>
>>> But introducing globally monotonous counter means that<br>
>>> there will be a single contention point.<br>
>><br>
>><br>
>> I wonder if the trade-off of Flake Ids [1] could be acceptable for this use<br>
>> case.<br>
>><br>
>> [1] <a href="http://yellerapp.com/posts/2015-02-09-flake-ids.html" rel="noreferrer" target="_blank">http://yellerapp.com/posts/<wbr>2015-02-09-flake-ids.html</a><br>
>><br>
>><br>
>> ______________________________<wbr>_________________<br>
>> infinispan-dev mailing list<br>
>> <a href="mailto:infinispan-dev@lists.jboss.org">infinispan-dev@lists.jboss.org</a><br>
>> <a href="https://lists.jboss.org/mailman/listinfo/infinispan-dev" rel="noreferrer" target="_blank">https://lists.jboss.org/<wbr>mailman/listinfo/infinispan-<wbr>dev</a><br>
> ______________________________<wbr>_________________<br>
> infinispan-dev mailing list<br>
> <a href="mailto:infinispan-dev@lists.jboss.org">infinispan-dev@lists.jboss.org</a><br>
> <a href="https://lists.jboss.org/mailman/listinfo/infinispan-dev" rel="noreferrer" target="_blank">https://lists.jboss.org/<wbr>mailman/listinfo/infinispan-<wbr>dev</a><br>
<br>
<br>
______________________________<wbr>_________________<br>
infinispan-dev mailing list<br>
<a href="mailto:infinispan-dev@lists.jboss.org">infinispan-dev@lists.jboss.org</a><br>
<a href="https://lists.jboss.org/mailman/listinfo/infinispan-dev" rel="noreferrer" target="_blank">https://lists.jboss.org/<wbr>mailman/listinfo/infinispan-<wbr>dev</a><br>
</div></div></blockquote></div><br></div></div>