[infinispan-dev] Infinispan and change data capture

[Gustavo Fernandes]

On Thu, Dec 15, 2016 at 2:53 PM, Emmanuel Bernard <emmanuel at hibernate.org>
wrote:

>
> On 15 Dec 2016, at 11:18, Gustavo Fernandes <gustavo at infinispan.org>
> wrote:
>
> On Thu, Dec 15, 2016 at 9:54 AM, Emmanuel Bernard <emmanuel at hibernate.org>
>  wrote:
>
>> The goal is as followed: allow to collect all changes to push them to
>> Debezium and thus Kafka.
>>
>> This need does not require to remember all changes since the beginning of
>> time in Infinispan. Just enough to:
>> - let Kafka catchup assuming it is the bottleneck
>> - let us not lose a change in Kafka when it happened in Infinispan
>> (coordinator, owner, replicas dying)
>>
>> The ability to read back history would then be handled by the Debezium /
>> Kafka tail, not infinispan itself.
>>
>>
> Having an embedded Debezium connector pushing everything to Kafka sounds
> cool, but what impact would it bring to the other stream consumers:
>
> * Remote listeners, which is supported in several clients apart from Java
> * Continuous Queries (the same)
> * Spark Stream
> * Other eventual 3rd party stream processors: Apache Flick, Storm, etc.
>
>
>
>
> Impact as in perf impact? Potential redesign impact? Or are you thinking
> of another question?
>


You mentioned that "The ability to read back history would then be handled
by the Debezium / Kafka tail, not infinispan itself", my question
was how the other consumers would get access to that history.



>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/infinispan-dev/attachments/20161215/5634b3f6/attachment.html