Infinispan and change data capture
by Randall Hauch
The Debezium project [1] is working on building change data capture connectors for a variety of databases. MySQL is available now, MongoDB will be soon, and PostgreSQL and Oracle are next on our roadmap.
One way in which Debezium and Infinispan can be used together is when Infinispan is being used as a cache for data stored in a database. In this case, Debezium can capture the changes to the database and produce a stream of events; a separate process can consume these change and evict entries from an Infinispan cache.
If Infinispan is to be used as a data store, then it would be useful for Debezium to be able to capture those changes so other apps/services can consume the changes. First of all, does this make sense? Secondly, if it does, then Debezium would need an Infinispan connector, and it’s not clear to me how that connector might capture the changes from Infinispan.
Debezium typically monitors the log of transactions/changes that are committed to a database. Of course how this works varies for each type of database. For example, MySQL internally produces a transaction log that contains information about every committed row change, and MySQL ensures that every committed change is included and that non-committed changes are excluded. The MySQL mechanism is actually part of the replication mechanism, so slaves update their internal state by reading the master’s log. The Debezium MySQL connector [2] simply reads the same log.
Infinispan has several mechanisms that may be useful:
Interceptors - See [3]. This seems pretty straightforward and IIUC provides access to all internal operations. However, it’s not clear to me whether a single interceptor will see all the changes in a cluster (perhaps in local and replicated modes) or only those changes that happen on that particular node (in distributed mode). It’s also not clear whether this interceptor is called within the context of the cache’s transaction, so if a failure happens just at the wrong time whether a change might be made to the cache but is not seen by the interceptor (or vice versa).
Cross-site replication - See [4][5]. A potential advantage of this mechanism appears to be that it is defined (more) globally, and it appears to function if the remote backup comes back online after being offline for a period of time.
State transfer - is it possible to participate as a non-active member of the cluster, and to effectively read all state transfer activities that occur within the cluster?
Cache store - tie into the cache store mechanism, perhaps by wrapping an existing cache store and sitting between the cache and the cache store
Monitor the cache store - don’t monitor Infinispan at all, and instead monitor the store in which Infinispan is storing entries. (This is probably the least attractive, since some stores can’t be monitored, or because the store is persisting an opaque binary value.)
Are there other mechanism that might be used?
There are a couple of important requirements for change data capture to be able to work correctly:
Upon initial connection, the CDC connector must be able to obtain a snapshot of all existing data, followed by seeing all changes to data that may have occurred since the snapshot was started. If the connector is stopped/fails, upon restart it needs to be able to reconnect and either see all changes that occurred since it last was capturing changes, or perform a snapshot. (Performing a snapshot upon restart is very inefficient and undesirable.) This works as follows: the CDC connector only records the “offset” in the source’s sequence of events; what this “offset” entails depends on the source. Upon restart, the connector can use this offset information to coordinate with the source where it wants to start reading. (In MySQL and PostgreSQL, every event includes the filename of the log and position in that file. MongoDB includes in each event the monotonically increasing timestamp of the transaction.
No change can be missed, even when things go wrong and components crash.
When a new entry is added, the “after” state of the entity will be included. When an entry is updated, the “after” state will be included in the event; if possible, the event should also include the “before” state. When an entry is removed, the “before” state should be included in the event.
Any thoughts or advice would be greatly appreciated.
Best regards,
Randall
[1] http://debezium.io
[2] http://debezium.io/docs/connectors/mysql/
[3] http://infinispan.org/docs/stable/user_guide/user_guide.html#_custom_inte...
[4] http://infinispan.org/docs/stable/user_guide/user_guide.html#CrossSiteRep...
[5] https://github.com/infinispan/infinispan/wiki/Design-For-Cross-Site-Repli...
8 years, 9 months
Spring module - change dependencies to Uber Jars
by Sebastian Laskawiec
Hey!
I'm currently trying to solve a tricky class loading issue connected to
Spring, CDI and Uber Jars. Here's the scenario:
- Remote Uber Jar contains CDI module
- Our Hot Rod client use newer version of JBoss Logging which is present
in Wildfly/EAP modules
- However EAP and Wildfly will load (and make available for deployment)
their own version of JBoss Logging [1]
- The easiest fix for this is to relocate JBoss Logging package in
Uber Jar
- Spring module requires some classes from Infinispan Common and they in
turn need BasicLogger from JBoss Logging
- If we relocate JBoss Logging and will try to use Uber Jar with
Spring - we will end up with classloading issue [2]
So it seems the best approach is to make Spring depend on Uber Jars instead
of "small ones". Of course, users who use small jars will probably be
affected by this change (they would have to either accept using Uber Jars
or exclude them in their poms and add dependencies manually).
Is anyone against this solution? JIRA tracking ticket: [3].
Thanks
Sebastian
[1] Scenario with Weld enabled WAR
https://docs.jboss.org/author/display/AS7/Implicit+module+dependencies+fo...
[2] https://bugzilla.redhat.com/show_bug.cgi?id=1266831#c7
[3] https://issues.jboss.org/browse/ISPN-6132
8 years, 10 months
Func API in tx cache
by Radim Vansa
Hi,
seems I'll have to implement the functional stuff on tx caches [1][2] if
I want to get rid of DeltaAware et al.
The general idea is quite simple - ReadOnly* commands should behave very
similar to non-tx mode, WriteOnly* commands will be just added as
modifications to the PrepareCommand and ReadWrite* commands will be both
added to modifications list, and sent to remote nodes where the result
won't be stored yet.
The results of operations should not be stored into transactional
context - the command will execute remotely (if the owners are remote)
unless the value was read by Get* beforehand.
With repeatable-reads isolation, the situation gets more complicated. If
we use ReadOnly* that performs identity lookup (effectively the same as
Get*) and the entry was modified in during the transaction, we can
return two different results - so a read committed semantics. With write
skew check enabled, we could at least fail the transaction at the end
(the check would be performed reads as well if the transaction contains
functional reads), but we cannot rely on WSC always on with RR.
Retrieving the whole entry and applying the functional command is not a
viable solution, IMO - that would completely defy the purpose of using
functional command.
A possible solution would be to send the global transaction ID with
those read commands and keep a remote transactional context with read
entries for the duration of transaction on remote nodes, too. However,
if we do a Read* command to primary owner, it's possible that further
Get* command will hit backup. So, we could go to all owners with Read*
already during the transaction (slowing down functional reads
considerably), or read only from primary owner (which slows down Get*s
even if we don't use functional APIs - this makes it a no-go). I am not
100% sure how a transaction transfer during ST will get into that.
We could also do it the ostrich way - "Yes we've promissed RR but Func
will be only RC". I'll probably do that in the first draft anyway.
Comments & opinions appreciated.
Radim
[1] https://issues.jboss.org/browse/ISPN-5806
[2] https://issues.jboss.org/browse/ISPN-6573
--
Radim Vansa <rvansa(a)redhat.com>
JBoss Performance Team
8 years, 11 months
if (trace) logger.tracef - it makes sense
by Sebastian Laskawiec
Hey!
A while ago I asked Radim and Dan about these kind of constructs [1]:
private boolean trace = logger.isTraceEnabled(); //stored in a field
... called in some method ...
if(trace)
logger.tracef(...);
...
At first they seemed wrong to me, because if one changes logging level
(using JMX for example), the code won't notice it. I also though it's quite
ok to use tracef directly, because JIT will inline and optimize it.
Unfortunately my benchmarks [2] show that I was wrong. Logger#tracef indeed
checks if the logging level is enabled but since JBoss Logging may use
different backends, the check is not trivial and is not inlined (at least
with default settings). The performance results look like this:
Benchmark Mode Cnt Score Error Units
MyBenchmark.noVariable thrpt 20 *717252060.124* ± 13420522.229 ops/s
MyBenchmark.withVariable thrpt 20 *2358360244.627* ± 50214969.572 ops/s
So if you even see a construct like this: logger.debuf or logger.tracef -
make sure you check if the logging level is enabled (and the check result
is stored in a field).
That was a bit surprising and interesting lesson :D
Thanks
Sebastian
[1] https://github.com/infinispan/infinispan/pull/4538#discussion_r80666086
[2] https://github.com/slaskawi/jboss-logging-perf-test
9 years
Hot Rod testing
by Tristan Tarrant
Recently I've had a chat with Galder, Will and Vittorio about how we
test the Hot Rod server module and the various clients. We also
discussed some of this in the past, but we now need to move forward with
a better strategy.
First up is the Hot Rod server module testsuite: it is the only part of
the code which still uses Scala. Will has a partial port of it to Java,
but we're wondering if it is worth completing that work, seeing that
most of the tests in that testsuite, in particular those related to the
protocol itself, are actually duplicated by the Java Hot Rod client's
testsuite which also happens to be our reference implementation of a
client and is much more extensive.
The only downside of removing it is that verification will require
running the client testsuite, instead of being self-contained.
Next up is how we test clients.
The Java client, partially described above, runs all of the tests
against ad-hoc embedded servers. Some of these tests, in particular
those related to topology, start and stop new servers on the fly.
The server integration testsuite performs yet another set of tests, some
of which overlap the above, but using the actual full-blown server. It
doesn't test for topology changes.
The C++ client wraps the native client in a Java wrapper generated by
SWIG and runs the Java client testsuite. It then checks against a
blacklist of known failures. It also has a small number of native tests
which use the server distribution.
The Node.js client has its own home-grown testsuite which also uses the
server distribution.
Duplication aside, which in some cases is unavoidable, it is impossible
to confidently say that each client is properly tested.
Since complete unification is impossible because of the different
testing harnesses used by the various platforms/languages, I propose the
following:
- we identify and group the tests depending on their scope (basic
protocol ops, bulk ops, topology/failover, security, etc). A client
which implements the functionality of a group MUST pass all of the tests
in that group with NO exceptions
- we assign a unique identifier to each group/test combination (e.g.
HR.BASIC.PUT, HR.BASIC.PUT_FLAGS_SKIP_LOAD, etc). These should be
collected in a "test book" (some kind of structured file) for comparison
with client test runs
- we refactor the Java client testsuite according to the above grouping
/ naming strategy so that testsuite which use the wrapping approach
(i.e. C++ with SWIG) can consume it by directly specifying the supported
groups
- other clients get reorganized so that they support the above grouping
I understand this is quite some work, but the current situation isn't
really sustainable.
Let me know what your thoughts are
Tristan
--
Tristan Tarrant
Infinispan Lead
JBoss, a division of Red Hat
9 years
Fine grained maps
by Radim Vansa
Hi all,
I have realized that fine grained maps don't work reliably with
write-skew check. This happens because WSC tries to load the entry from
DC/cache-store, compare versions and store it, assuming that this
happens atomically as the entry is locked. However, as fine grained maps
can lock two different keys and modify the same entry, there is a risk
that the check & store won't be atomic. Right now, the update itself
won't be lost, because fine grained maps use DeltaAwareCacheEntries
which apply the updates DC's lock (there can be some problems when
passivation is used, though, [1] hopefully deals with them).
I have figured this out when trying to update the DeltaAware handling to
support more than just atomic maps - yes, there are special branches for
atomic maps in the code, which is quite ugly design-wise, IMO. My
intention is to do similar things like WSC for replaying the deltas, but
this, obviously, needs some atomicity.
IIUC, fine-grained locking was introduced back in 5.1 because of
deadlocks in the lock-acquisition algorithm; the purpose was not to
improve concurrency. Luckily, the days of deadlocks are far back, now we
can get the cluster stuck in more complex ways :) Therefore, with a
correctness-first approach, in optimistic caches I would lock just the
main key (not the composite keys). The prepare-commit should be quite
fast anyway, and I don't see how this could affect users
(counter-examples are welcome) but slightly reduced concurrency.
In pessimistic caches we have to be more cautious, since users
manipulate the locks directly and reason about them more. Therefore, we
need to lock the composite keys during transaction runtime, but in
addition to that, during the commit itself we should lock the main key
for the duration of the commit if necessary - pessimistic caches don't
sport WSC, but I was looking for some atomicity options for deltas.
An alternative would be to piggyback on DC's locking scheme, however,
this is quite unsuitable for the optimistic case with a RPC between WSC
and DC store. In addition to that, it doesn't fit into our async picture
and we would send complex compute functions into the DC, instead of
decoupled lock/unlock. I could also devise another layer of locking, but
that's just madness.
I am adding Sanne to recipients as OGM is probably the most important
consumer of atomic hash maps.
WDYT?
Radim
[1]
https://github.com/infinispan/infinispan/pull/4564/commits/2eeb7efbd4e1ea...
--
Radim Vansa <rvansa(a)redhat.com>
JBoss Performance Team
9 years
Documentation mini-sprint
by Tristan Tarrant
Hi all,
I have created a PR [1] which contains a TODO list for our
documentation. This makes the review process public and uses the
familiar GitHub tooling.
I would like everybody to dedicate some time before the end of this week
in looking at the items and making proposals to improve the docs.
On Friday I will merge the TODO list to master.
Starting on Monday we will dedicate 3 days on doing as much as possible
to reduce that list to zero. Documentation PRs will need to remove the
corresponding lines from the TODO list.
Thanks
Tristan
[1] https://github.com/infinispan/infinispan/pull/4572
--
Tristan Tarrant
Infinispan Lead
JBoss, a division of Red Hat
9 years
To which issue do I assocate my PR work for a quickstart update?
by Martes Wigglesworth
Greetings all.
For the quickstart PR work, what issue should I reference, when the work is done?
The relevant information on each issue is listed below, as was updated by jbossbot, when posting to infinispan channel on freenode.
(2:24:44 PM) jbossbot: jira [ ISPN-7037 ] Create quickstart that demonstrates library mode node security using callbackhandlers [ New (Unresolved) Feature Request , Minor , Demos and Tutorials/Tasks , Martes Wigglesworth ] https://issues.jboss.org/browse/ISPN-7037
(2:24:44 PM) jbossbot: jira [ JDG-523 ] An updated secure library mode quickstart should be created which demonstrates node security using callbackhandlers for authenticating new entrants to a jgroups cluster. [ New (Unresolved) Enhancement , Minor , unspecified , Tristan Tarrant ] https://issues.jboss.org/browse/JDG-523
Respectfully,
Martes G Wigglesworth
Consultant - Middleware Engineer
Red Hat Consulting
Red Hat, Inc.
Office Phone: 804 343 6084 - 8136084
Office Email: mwiggles(a)redhat.com
9 years
