<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><br class=""><div><blockquote type="cite" class=""><div class="">On 15 Dec 2016, at 15:59, Gustavo Fernandes <<a href="mailto:gustavo@infinispan.org" class="">gustavo@infinispan.org</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><div class="gmail_extra"><br class="Apple-interchange-newline"><br class=""><div class="gmail_quote">On Thu, Dec 15, 2016 at 2:53 PM, Emmanuel Bernard<span class="Apple-converted-space"> </span><span dir="ltr" class=""><<a href="mailto:emmanuel@hibernate.org" target="_blank" class="">emmanuel@hibernate.org</a>></span><span class="Apple-converted-space"> </span>wrote:<br class=""><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div style="overflow-wrap: break-word;" class=""><span class="gmail-"><br class=""><div class=""><blockquote type="cite" class=""><div class="">On 15 Dec 2016, at 11:18, Gustavo Fernandes <<a href="mailto:gustavo@infinispan.org" target="_blank" class="">gustavo@infinispan.org</a>> wrote:</div><br class="gmail-m_-119706947963337183Apple-interchange-newline"><div class=""><div dir="ltr" style="font-family: helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;" class="">On Thu, Dec 15, 2016 at 9:54 AM, Emmanuel Bernard<span class="gmail-m_-119706947963337183Apple-converted-space"> </span><span dir="ltr" class=""><<a href="mailto:emmanuel@hibernate.org" target="_blank" class="">emmanuel@hibernate.<wbr class="">org</a>></span><span class="gmail-m_-119706947963337183Apple-converted-space"> </span>wrote:<br class=""><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;">The goal is as followed: allow to collect all changes to push them to Debezium and thus Kafka.<br class=""><br class="">This need does not require to remember all changes since the beginning of time in Infinispan. Just enough to:<br class="">- let Kafka catchup assuming it is the bottleneck<br class="">- let us not lose a change in Kafka when it happened in Infinispan (coordinator, owner, replicas dying)<br class=""><br class="">The ability to read back history would then be handled by the Debezium / Kafka tail, not infinispan itself.<br class=""><br class=""></blockquote><div class=""><br class=""></div><div class="">Having an embedded Debezium connector pushing everything to Kafka sounds cool, but what impact would it bring to the other stream consumers:<br class=""><br class="">* Remote listeners, which is supported in several clients apart from Java<br class=""></div><div class="">* Continuous Queries (the same)<br class=""></div><div class="">* Spark Stream<br class=""></div><div class="">* Other eventual 3rd party stream processors: Apache Flick, Storm, etc.<br class=""><br class=""></div><div class=""> </div></div></div></div></div></blockquote><br class=""></div></span><div class="">Impact as in perf impact? Potential redesign impact? Or are you thinking of another question?</div></div></blockquote><div class=""><br class=""><br class=""></div><div class="">You mentioned that "<span class="gmail-">The ability to read back history would then be handled by the Debezium / Kafka tail, not infinispan itself", my question<br class=""></span></div><div class=""><span class="gmail-">was how the other consumers would get access to that history.<br class=""></span></div></div></div></div></div></blockquote><br class=""></div><div>Yes that’s an interesting point.</div><div><br class=""></div><div>First off here we are describing an ad-hoc model where we push changes to Debezium and then Kafka.</div><div>But the underlying temp queue mechanism I described on the Dec 9th email might be used to harden the code pushing changes to the sources you describe and that even improve the continuous queries engine and the Spark DStream integration I suppose.</div><div>Maybe we want a more generic mechanism relying on that temp queue system to plug a list of consumers. And focus on Spark Stream, Continuous queries and Debezium as a first set of “clients”.</div><div><br class=""></div><div>For the ability to read back in history, I am happy to force consumers to go through a Kafka queue. As others pointed out, if we make Infinispan a durable queue system, we are making a different Infinispan than what it is today and this is probably undesirable.</div><div><br class=""></div><div>Emmanuel</div><div><br class=""></div><br class=""></body></html>