<html><head><meta http-equiv="Content-Type" content="text/html charset=windows-1252"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><br class=""><div><blockquote type="cite" class=""><div class="">On Jul 11, 2016, at 3:42 AM, Adrian Nistor <<a href="mailto:anistor@redhat.com" class="">anistor@redhat.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class="">
<meta content="text/html; charset=windows-1252" http-equiv="Content-Type" class="">
<div bgcolor="#FFFFFF" text="#000000" class="">
<div class="moz-cite-prefix">Hi Randall,<br class="">
<br class="">
Infinispan supports both push and pull access models. The push
model is supported by events (and listeners), which are cluster
wide and are available in both library and remote mode (hotrod).
The notification system is pretty advanced as there is a filtering
mechanism available that can use a hand coded filter / converter
or one specified in jpql (experimental atm). Getting a snapshot of
the initial data is also possible. But infinispan does not produce
a transaction log to be used for determining all changes that
happened since a previous connection time, so you'll always have
to get a new full snapshot when re-connecting. <br class="">
<br class="">
So if Infinispan is the data store I would base the Debezium
connector implementation on Infinispan's event notification
system. Not sure about the other use case though.<br class="">
<br class=""></div></div></div></blockquote><div><br class=""></div><div>Thanks, Adrian, for the feedback. A couple of questions.</div><div><br class=""></div><div>You mentioned Infinispan has a pull model — is this just using the normal API to read the entries?</div><div><br class=""></div><div>With event listeners, a single connection will receive all of the events that occur in the cluster, correct? Is it possible (e.g., a very unfortunately timed crash) for a change to be made to the cache without an event being produced and sent to listeners? What happens if the network fails or partitions? How does cross site replication address this?</div><div><br class=""></div><div>Has there been any thought about adding to Infinispan a write ahead log or transaction log to each node or, better yet, for the whole cluster?</div><div><div class=""><br class=""></div></div>Thanks again!</div><div><br class=""><blockquote type="cite" class=""><div class=""><div bgcolor="#FFFFFF" text="#000000" class=""><div class="moz-cite-prefix">
Adrian<br class="">
<br class="">
On 07/09/2016 04:38 PM, Randall Hauch wrote:<br class="">
</div>
<blockquote cite="mid:2569A6BA-FBC2-40A7-A821-26676F10BEB0@redhat.com" type="cite" class="">
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252" class="">
The Debezium project [1] is working on building change data
capture connectors for a variety of databases. MySQL is available
now, MongoDB will be soon, and PostgreSQL and Oracle are next on
our roadmap.
<div class=""><br class="">
</div>
<div class="">One way in which Debezium and Infinispan can be used
together is when Infinispan is being used as a cache for data
stored in a database. In this case, Debezium can capture the
changes to the database and produce a stream of events; a
separate process can consume these change and evict entries from
an Infinispan cache.</div>
<div class=""><br class="">
</div>
<div class="">If Infinispan is to be used as a data store, then it
would be useful for Debezium to be able to capture those changes
so other apps/services can consume the changes. First of all,
does this make sense? Secondly, if it does, then Debezium would
need an Infinispan connector, and it’s not clear to me how that
connector might capture the changes from Infinispan.<br class="">
<div class=""><br class="">
</div>
<div class="">Debezium typically monitors the log of
transactions/changes that are committed to a database. Of
course how this works varies for each type of database. For
example, MySQL internally produces a transaction log that
contains information about every committed row change, and
MySQL ensures that every committed change is included and that
non-committed changes are excluded. The MySQL mechanism is
actually part of the replication mechanism, so slaves update
their internal state by reading the master’s log. The Debezium
MySQL connector [2] simply reads the same log.</div>
<div class=""><br class="">
</div>
<div class="">Infinispan has several mechanisms that may be
useful:</div>
<div class=""><br class="">
</div>
<div class="">
<ul class="MailOutline">
<li class="">Interceptors - See [3]. This seems pretty
straightforward and IIUC provides access to all internal
operations. However, it’s not clear to me whether a single
interceptor will see all the changes in a cluster (perhaps
in local and replicated modes) or only those changes that
happen on that particular node (in distributed mode). It’s
also not clear whether this interceptor is called within
the context of the cache’s transaction, so if a failure
happens just at the wrong time whether a change might be
made to the cache but is not seen by the interceptor (or
vice versa).</li>
<li class="">Cross-site replication - See [4][5]. A
potential advantage of this mechanism appears to be that
it is defined (more) globally, and it appears to function
if the remote backup comes back online after being offline
for a period of time.</li>
<li class="">State transfer - is it possible to participate
as a non-active member of the cluster, and to effectively
read all state transfer activities that occur within the
cluster?</li>
<li class="">Cache store - tie into the cache store
mechanism, perhaps by wrapping an existing cache store and
sitting between the cache and the cache store</li>
<li class="">Monitor the cache store - don’t monitor
Infinispan at all, and instead monitor the store in which
Infinispan is storing entries. (This is probably the least
attractive, since some stores can’t be monitored, or
because the store is persisting an opaque binary value.)</li>
</ul>
<div class=""><br class="">
</div>
</div>
<div class="">Are there other mechanism that might be used?</div>
<div class=""><br class="">
</div>
<div class="">There are a couple of important requirements for
change data capture to be able to work correctly:</div>
<div class=""><br class="">
</div>
<div class="">
<ol class="MailOutline">
<li class="">Upon initial connection, the CDC connector must
be able to obtain a snapshot of all existing data,
followed by seeing all changes to data that may have
occurred since the snapshot was started. If the connector
is stopped/fails, upon restart it needs to be able to
reconnect and either see all changes that occurred since
it last was capturing changes, or perform a snapshot.
(Performing a snapshot upon restart is very inefficient
and undesirable.) This works as follows: the CDC connector
only records the “offset” in the source’s sequence of
events; what this “offset” entails depends on the source.
Upon restart, the connector can use this offset
information to coordinate with the source where it wants
to start reading. (In MySQL and PostgreSQL, every event
includes the filename of the log and position in that
file. MongoDB includes in each event the monotonically
increasing timestamp of the transaction.</li>
<li class="">No change can be missed, even when things go
wrong and components crash.</li>
<li class="">When a new entry is added, the “after” state of
the entity will be included. When an entry is updated, the
“after” state will be included in the event; if possible,
the event should also include the “before” state. When an
entry is removed, the “before” state should be included in
the event.</li>
</ol>
</div>
<div class=""><br class="">
</div>
<div class="">Any thoughts or advice would be greatly
appreciated.</div>
<div class=""><br class="">
</div>
<div class="">Best regards,</div>
<div class=""><br class="">
</div>
<div class="">Randall</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class="">[1] <a moz-do-not-send="true" href="http://debezium.io/" class="">http://debezium.io</a></div>
<div class="">[2] <a moz-do-not-send="true" href="http://debezium.io/docs/connectors/mysql/" class="">http://debezium.io/docs/connectors/mysql/</a></div>
<div class="">[3] <a moz-do-not-send="true" href="http://infinispan.org/docs/stable/user_guide/user_guide.html#_custom_interceptors_chapter" class="">http://infinispan.org/docs/stable/user_guide/user_guide.html#_custom_interceptors_chapter</a></div>
<div class="">[4] <a moz-do-not-send="true" href="http://infinispan.org/docs/stable/user_guide/user_guide.html#CrossSiteReplication" class="">http://infinispan.org/docs/stable/user_guide/user_guide.html#CrossSiteReplication</a></div>
<div class="">[5] <a moz-do-not-send="true" href="https://github.com/infinispan/infinispan/wiki/Design-For-Cross-Site-Replication" class="">https://github.com/infinispan/infinispan/wiki/Design-For-Cross-Site-Replication</a></div>
</div>
<br class="">
<fieldset class="mimeAttachmentHeader"></fieldset>
<br class="">
<pre wrap="" class="">_______________________________________________
infinispan-dev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:infinispan-dev@lists.jboss.org">infinispan-dev@lists.jboss.org</a>
<a class="moz-txt-link-freetext" href="https://lists.jboss.org/mailman/listinfo/infinispan-dev">https://lists.jboss.org/mailman/listinfo/infinispan-dev</a></pre>
</blockquote>
<br class="">
</div>
</div></blockquote></div><br class=""></body></html>