[infinispan-dev] Infinispan and change data capture

Thu Dec 15 11:02:23 EST 2016

> On Dec 15, 2016, at 9:09 AM, Sanne Grinovero <sanne at infinispan.org> wrote:
> 
> Thanks Randall,
> those clarifications have been great.
> 
> Emmanuel: some of your statements conflict with Randall's
> clarifications and with the feasibility points I've been pointing at.
> You say "collect *all* changes". I've been questioning that Infinispan
> can not keep *all* changes around for a given single key; I understand
> we'd allow clients to retrieve streams of changes persisted into
> Kafka, but we need to be clear that we won't be handling *all* changes
> to Kafka (nor to Debezium), so the magic these can do is somewhat
> limited. They can certainly expand on the capabilities that Infinispan
> would provide on its own, but some of the use cases which Gustavo
> mentioned would not be suitable.

I’m not sure we were saying conflicting things. I was saying what is possible: Debezium would capture whatever it can from Infinispan via a client listener API and record it as a stream of change events. I think Emmanuel was arguing that the stream will be (far?) more useful to a wider range of consumers if it has every change made by Infinispan, compared to a stream that contains only some of the changes made in Infinispan.

> 
> I don't think this is a big problem in practice though; take the
> example of monitoring fluctuations of value of some stock symbol for
> example: it wouldn't be possible to investigate derivative numbers
> from these fluctuations just from the Key/Value pair "stock name" /
> "value", however people can store such events in a different way, for
> example by using a composite key "stock name" + "timestamp". People
> just need clarity on how this works, including us to model the storage
> appropriately.
> 
> Thanks,
> Sanne
> 
> 
> On 15 December 2016 at 09:54, Emmanuel Bernard <emmanuel at hibernate.org> wrote:
>> The goal is as followed: allow to collect all changes to push them to Debezium and thus Kafka.
>> 
>> This need does not require to remember all changes since the beginning of time in Infinispan. Just enough to:
>> - let Kafka catchup assuming it is the bottleneck
>> - let us not lose a change in Kafka when it happened in Infinispan (coordinator, owner, replicas dying)
>> 
>> The ability to read back history would then be handled by the Debezium / Kafka tail, not infinispan itself.
>> 
>> Check my email on this tread from Dec 9th.
>> 
>>> On 12 Dec 2016, at 16:13, Sanne Grinovero <sanne at infinispan.org> wrote:
>>> 
>>> I'm reading many clever suggestions for various aspects of such a
>>> system, but I fail to see a clear definition of the goal.
>>> 
>>>> From Randall's opening email I understand how MySQL does this, but
>>> it's an example and I'm not sure which aspects are implementation
>>> details of how MySQL happens to accomplish this, and which aspects are
>>> requirements for the Infinispan enhancement proposals.
>>> 
>>> I remember a meeting with Manik Surtani, Jonathan Halliday and Mark
>>> Little, whose outcome was a general agreement that Infinispan would
>>> eventually need both tombstones and versioned entries, not just for
>>> change data capture but to improve several other aspects;
>>> unfortunately that was in December 2010 and never became a priority,
>>> but the benefits are clear.
>>> The complexities which have put off such plans lie in the "garbage
>>> collection", aka the need to not grow the history without bounds, and
>>> have to drop or compact history.
>>> 
>>> So I'm definitely sold on the need to add a certain amount of history,
>>> but we need to define how much of this history is expected to be held.
>>> 
>>> In short, what's the ultimate goal? I see two main but different
>>> options intertwined:
>>> - allow to synchronize the *final state* of a replica
>>> - inspect specific changes
>>> 
>>> For the first case, it would be enough for us to be able to provide a
>>> "squashed history" (as in Git squash), but we'd need to keep versioned
>>> shapshots around and someone needs to tell you which ones can be
>>> garbage collected.
>>> For example when a key is: written, updated, updated, deleted since
>>> the snapshot, we'll send only "deleted" as the intermediary states are
>>> irrelevant.
>>> For the second case, say the goal is to inspect fluctuations of price
>>> variations of some item, then the intermediary states are not
>>> irrelevant.
>>> 
>>> Which one will we want to solve? Both?
>>> Personally the attempt of solving the second one seems like a huge
>>> pivot of the project, the current data-structures and storage are not
>>> designed for this. I see the value of such benefits, but maybe
>>> Infinispan is not the right tool for such a problem.
>>> 
>>> I'd prefer to focus on the benefits of the squashed history, and have
>>> versioned entries soon, but even in that case we need to define which
>>> versions need to be kept around, and how garbage collection /
>>> vacuuming is handled.
>>> This can be designed to be transparent to the client: handled as an
>>> internal implementation detail which we use to improve performance of
>>> Infinispan itself, or it can be exposed to clients to implement change
>>> data capture, but in this case we need to track which clients are
>>> still going to need an older snapshot; this has an impact as clients
>>> would need to be registered, and has a significant impact on the
>>> storage strategies.
>>> 
>>> Within Kafka the log compaction strategies are configurable; I have no
>>> experience with Kafka but the docs seem to suggest that it's most
>>> often used to provide the last known value of each key. That would be
>>> doable for us, but Kafka also does allow optionally for wider scope
>>> retention strategies: can we agree that that would not be an option
>>> with Infinispan? If not, these goals need to be clarified.
>>> 
>>> My main concern is that if we don't limit the scope of which
>>> capabilities we want Infinispan to provide, it risks to become the
>>> same thing as Kafka, rather than integrating with it. I don't think we
>>> want to pivot all our storage design into efficiently treating large
>>> scale logs.
>>> 
>>> In short, I'd like to see an agreement that analyzing e.g.
>>> fluctuations in stock prices would be a non-goal, if these are stored
>>> as {"stock name", value} key/value pairs. One could still implement
>>> such a thing by using a more sophisticated model, just don't expect to
>>> be able to see all intermediary values each entry has ever had since
>>> the key was first used.
>>> 
>>> # Commenting on specific proposals
>>> 
>>> On ID generation: I'd definitely go with IDs per segment rather than
>>> IDs per key for the purpose of change data capture. If you go with
>>> independent IDs per key, the client would need to keep track of each
>>> version of each entry, which has an high overhead and degree of
>>> complexity for the clients.
>>> On the other hand, we already guarantee that each segment is managed
>>> by a single primary owner, so attaching the "segment transaction id"
>>> to each internal entry being changed can be implemented efficiently by
>>> Infinispan.
>>> Segment ownership handoff needs to be highly consistent during cluster
>>> topology changes, but that requirement already exists; we'd just need
>>> to make sure that this monotonic counter is included during the
>>> handoff of the responsibility as primary owner of a segment.
>>> 
>>> Thanks,
>>> 
>>> Sanne
>>> 
>>> 
>>> 
>>> 
>>> 
>>> On 12 December 2016 at 13:58, Gustavo Fernandes <gustavo at infinispan.org> wrote:
>>>> 
>>>> 
>>>> On Fri, Dec 9, 2016 at 9:13 AM, Radim Vansa <rvansa at redhat.com> wrote:
>>>>> 
>>>>> But introducing globally monotonous counter means that
>>>>> there will be a single contention point.
>>>> 
>>>> 
>>>> I wonder if the trade-off of Flake Ids [1] could be acceptable for this use
>>>> case.
>>>> 
>>>> [1] http://yellerapp.com/posts/2015-02-09-flake-ids.html
>>>> 
>>>> 
>>>> _______________________________________________
>>>> infinispan-dev mailing list
>>>> infinispan-dev at lists.jboss.org
>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>> _______________________________________________
>>> infinispan-dev mailing list
>>> infinispan-dev at lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>> 
>> 
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev