On 15 Dec 2016, at 11:18, Gustavo Fernandes <gustavo@infinispan.org> wrote:

On Thu, Dec 15, 2016 at 9:54 AM, Emmanuel Bernard <emmanuel@hibernate.org> wrote:

The goal is as followed: allow to collect all changes to push them to Debezium and thus Kafka.

This need does not require to remember all changes since the beginning of time in Infinispan. Just enough to:
- let Kafka catchup assuming it is the bottleneck
- let us not lose a change in Kafka when it happened in Infinispan (coordinator, owner, replicas dying)

The ability to read back history would then be handled by the Debezium / Kafka tail, not infinispan itself.

Having an embedded Debezium connector pushing everything to Kafka sounds cool, but what impact would it bring to the other stream consumers:

* Remote listeners, which is supported in several clients apart from Java

* Continuous Queries (the same)

* Spark Stream

* Other eventual 3rd party stream processors: Apache Flick, Storm, etc.

 

Impact as in perf impact? Potential redesign impact? Or are you thinking of another question?

_______________________________________________
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev