Infinispan and change data capture
by Randall Hauch
The Debezium project [1] is working on building change data capture connectors for a variety of databases. MySQL is available now, MongoDB will be soon, and PostgreSQL and Oracle are next on our roadmap.
One way in which Debezium and Infinispan can be used together is when Infinispan is being used as a cache for data stored in a database. In this case, Debezium can capture the changes to the database and produce a stream of events; a separate process can consume these change and evict entries from an Infinispan cache.
If Infinispan is to be used as a data store, then it would be useful for Debezium to be able to capture those changes so other apps/services can consume the changes. First of all, does this make sense? Secondly, if it does, then Debezium would need an Infinispan connector, and it’s not clear to me how that connector might capture the changes from Infinispan.
Debezium typically monitors the log of transactions/changes that are committed to a database. Of course how this works varies for each type of database. For example, MySQL internally produces a transaction log that contains information about every committed row change, and MySQL ensures that every committed change is included and that non-committed changes are excluded. The MySQL mechanism is actually part of the replication mechanism, so slaves update their internal state by reading the master’s log. The Debezium MySQL connector [2] simply reads the same log.
Infinispan has several mechanisms that may be useful:
Interceptors - See [3]. This seems pretty straightforward and IIUC provides access to all internal operations. However, it’s not clear to me whether a single interceptor will see all the changes in a cluster (perhaps in local and replicated modes) or only those changes that happen on that particular node (in distributed mode). It’s also not clear whether this interceptor is called within the context of the cache’s transaction, so if a failure happens just at the wrong time whether a change might be made to the cache but is not seen by the interceptor (or vice versa).
Cross-site replication - See [4][5]. A potential advantage of this mechanism appears to be that it is defined (more) globally, and it appears to function if the remote backup comes back online after being offline for a period of time.
State transfer - is it possible to participate as a non-active member of the cluster, and to effectively read all state transfer activities that occur within the cluster?
Cache store - tie into the cache store mechanism, perhaps by wrapping an existing cache store and sitting between the cache and the cache store
Monitor the cache store - don’t monitor Infinispan at all, and instead monitor the store in which Infinispan is storing entries. (This is probably the least attractive, since some stores can’t be monitored, or because the store is persisting an opaque binary value.)
Are there other mechanism that might be used?
There are a couple of important requirements for change data capture to be able to work correctly:
Upon initial connection, the CDC connector must be able to obtain a snapshot of all existing data, followed by seeing all changes to data that may have occurred since the snapshot was started. If the connector is stopped/fails, upon restart it needs to be able to reconnect and either see all changes that occurred since it last was capturing changes, or perform a snapshot. (Performing a snapshot upon restart is very inefficient and undesirable.) This works as follows: the CDC connector only records the “offset” in the source’s sequence of events; what this “offset” entails depends on the source. Upon restart, the connector can use this offset information to coordinate with the source where it wants to start reading. (In MySQL and PostgreSQL, every event includes the filename of the log and position in that file. MongoDB includes in each event the monotonically increasing timestamp of the transaction.
No change can be missed, even when things go wrong and components crash.
When a new entry is added, the “after” state of the entity will be included. When an entry is updated, the “after” state will be included in the event; if possible, the event should also include the “before” state. When an entry is removed, the “before” state should be included in the event.
Any thoughts or advice would be greatly appreciated.
Best regards,
Randall
[1] http://debezium.io
[2] http://debezium.io/docs/connectors/mysql/
[3] http://infinispan.org/docs/stable/user_guide/user_guide.html#_custom_inte...
[4] http://infinispan.org/docs/stable/user_guide/user_guide.html#CrossSiteRep...
[5] https://github.com/infinispan/infinispan/wiki/Design-For-Cross-Site-Repli...
8 years
Spring module - change dependencies to Uber Jars
by Sebastian Laskawiec
Hey!
I'm currently trying to solve a tricky class loading issue connected to
Spring, CDI and Uber Jars. Here's the scenario:
- Remote Uber Jar contains CDI module
- Our Hot Rod client use newer version of JBoss Logging which is present
in Wildfly/EAP modules
- However EAP and Wildfly will load (and make available for deployment)
their own version of JBoss Logging [1]
- The easiest fix for this is to relocate JBoss Logging package in
Uber Jar
- Spring module requires some classes from Infinispan Common and they in
turn need BasicLogger from JBoss Logging
- If we relocate JBoss Logging and will try to use Uber Jar with
Spring - we will end up with classloading issue [2]
So it seems the best approach is to make Spring depend on Uber Jars instead
of "small ones". Of course, users who use small jars will probably be
affected by this change (they would have to either accept using Uber Jars
or exclude them in their poms and add dependencies manually).
Is anyone against this solution? JIRA tracking ticket: [3].
Thanks
Sebastian
[1] Scenario with Weld enabled WAR
https://docs.jboss.org/author/display/AS7/Implicit+module+dependencies+fo...
[2] https://bugzilla.redhat.com/show_bug.cgi?id=1266831#c7
[3] https://issues.jboss.org/browse/ISPN-6132
8 years, 1 month
Multi tenancy support for Infinispan
by Sebastian Laskawiec
Dear Community,
Please have a look at the design of Multi tenancy support for Infinispan
[1]. I would be more than happy to get some feedback from you.
Highlights:
- The implementation will be based on a Router (which will be built
based on Netty)
- Multiple Hot Rod and REST servers will be attached to the router which
in turn will be attached to the endpoint
- The router will operate on a binary protocol when using Hot Rod
clients and path-based routing when using REST
- Memcached will be out of scope
- The router will support SSL+SNI
Thanks
Sebastian
[1]
https://github.com/infinispan/infinispan/wiki/Multi-tenancy-for-Hotrod-Se...
8 years, 3 months
Unwrapping exceptions
by Radim Vansa
When enhancing functional API implementation, I've stumbled upon the way
the exceptions from lambdas should be thrown to user code.
Intuitively I've expected that when my lambda throws an exception, I'll
catch it from cache.eval(...). That may be the case when the lambda is
still executed locally, but when it goes remote, the exception is
wrapped into RemoteException etc.
In the past, exceptions being thrown were a matter of failure inside
Infinispan, but what about here? User does not need to know where the
functional expression has been executed, so it seems to me that he
should get the exception directly.
WDYT?
Radim
--
Radim Vansa <rvansa(a)redhat.com>
JBoss Performance Team
8 years, 4 months
Optional in listener events
by Radim Vansa
Hi,
as I am trying to simplify current entry wrapping and distribution code,
I often find that listeners can get wrong previous value in the event,
and it sometimes forces the command to load the value even if it is not
needed for the command.
I am wondering if we should change the previous value in events to
Optional - we can usually at least detect that we cannot provide a
reliable value (e.g. after retry due to topology change, or because the
command did not bothered to load the previous value from cache loader)
and return empty Optional.
WDYT?
Radim
--
Radim Vansa <rvansa(a)redhat.com>
JBoss Performance Team
8 years, 4 months
JGroups Error: JGRP000029 observed with "JGroups 3.6.8 + Infinispan 8.2.2":
by Manohar SL
Hi Bela Ban,
We have been trying to use JGroups 3.6.8 with Infinispan 8.2.2., in this context we are observing an issue with the Replication Mode of Infinispan usage.
We see the below exception from Jgroups:
JGRP000029: failed sending message to <>:7800 (100 bytes): java.lang.NullPointerException, headers: TCPGOSSIP: [type=GET_MBRS_REQ, cluster=x-cluster], TP: [cluster_name=x-cluster]
The configurations used are highlighted below:
Infinispan config
<infinispan>
<jgroups>
<stack-file name="configurationFile" path="config/jgroups.xml"/>
</jgroups>
<cache-container>
<transport cluster="x-cluster" stack="configurationFile" />
<replicated-cache name="transactional-type" mode="SYNC">
<transaction mode="NON_XA" locking="OPTIMISTIC" transaction-manager-lookup="org.infinispan.transaction.lookup.JBossStandaloneJTAManagerLookup" auto-commit="true" />
<locking acquire-timeout="60000"/>
<expiration lifespan="43200000"/>
</replicated-cache>
</cache-container>
</infinispan>
Jgroups configuration
<!--
TCP based stack, with flow control and message bundling. This is usually used when IP
multicasting cannot be used in a network, e.g. because it is disabled (routers discard multicast).
Note that TCP.bind_addr and TCPPING.initial_hosts should be set, possibly via system properties, e.g.
-Djgroups.bind_addr=192.168.5.2 and -Djgroups.tcpping.initial_hosts=192.168.5.2[7800]".
author: Bela Ban
-->
<config xmlns="urn:org:jgroups"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:org:jgroups http://www.jgroups.org/schema/jgroups-3.6.xsd">
<TCP loopback="true"
bind_addr="${jgroups.tcp.address:127.0.0.1}"
bind_port="${jgroups.tcp.port:7800}"
recv_buf_size="${tcp.recv_buf_size:20M}"
send_buf_size="${tcp.send_buf_size:640K}"
discard_incompatible_packets="true"
max_bundle_size="64K"
max_bundle_timeout="30"
enable_bundling="true"
use_send_queues="true"
sock_conn_timeout="300"
timer_type="new"
timer.min_threads="4"
timer.max_threads="10"
timer.keep_alive_time="3000"
timer.queue_max_size="500"
thread_pool.enabled="true"
thread_pool.min_threads="2"
thread_pool.max_threads="30"
thread_pool.keep_alive_time="60000"
thread_pool.queue_enabled="false"
thread_pool.queue_max_size="100"
thread_pool.rejection_policy="discard"
oob_thread_pool.enabled="true"
oob_thread_pool.min_threads="2"
oob_thread_pool.max_threads="30"
oob_thread_pool.keep_alive_time="60000"
oob_thread_pool.queue_enabled="false"
oob_thread_pool.queue_max_size="100"
oob_thread_pool.rejection_policy="discard"/>
<!-- <TCP_NIO -->
<!-- bind_port="7800" -->
<!-- bind_interface="${jgroups.tcp_nio.bind_interface:bond0}" -->
<!-- use_send_queues="true" -->
<!-- sock_conn_timeout="300" -->
<!-- reader_threads="3" -->
<!-- writer_threads="3" -->
<!-- processor_threads="0" -->
<!-- processor_minThreads="0" -->
<!-- processor_maxThreads="0" -->
<!-- processor_queueSize="100" -->
<!-- processor_keepAliveTime="9223372036854775807"/> -->
<TCPGOSSIP initial_hosts="${jgroups.tcpgossip.initial_hosts}"/>
<!-- <TCPPING async_discovery="true" initial_hosts="${jgroups.tcpping.initial_hosts}"
port_range="2" timeout="3000" /> -->
<MERGE2 max_interval="30000" min_interval="10000"/>
<FD_SOCK/>
<FD timeout="3000" max_tries="3"/>
<VERIFY_SUSPECT timeout="1500"/>
<pbcast.NAKACK
use_mcast_xmit="false"
retransmit_timeout="300,600,1200,2400,4800"
discard_delivered_msgs="false"/>
<UNICAST2 timeout="300,600,1200"
stable_interval="5000"
max_bytes="1m"/>
<pbcast.STABLE stability_delay="500" desired_avg_gossip="5000" max_bytes="1m"/>
<pbcast.GMS print_local_addr="false" join_timeout="3000" view_bundling="true"/>
<UFC max_credits="200k" min_threshold="0.20"/>
<MFC max_credits="200k" min_threshold="0.20"/>
<FRAG2 frag_size="60000"/>
<RSVP timeout="60000" resend_interval="500" ack_on_delivery="false" />
</config>
Any help on this would be really great.
Kindly let us know if you would need any further information on this.
Regs,
Manohar.
8 years, 4 months
Infinispan and OpenShift/Kubernetes PetSets
by Sebastian Laskawiec
Hey!
I've been playing with Kubernetes PetSets [1] for a while and I'd like to
share some thoughts. Before I dig in, let me give you some PetSets
highlights:
- PetSets are alpha resources for managing stateful apps in Kubernetes
1.3 (and OpenShift Origin 1.3).
- Since this is an alpha resource, there are no guarantees about
backwards compatibility. Alpha resources can also be disabled in some
public cloud providers (you can control which API versions are accessible
[2]).
- PetSets allows starting pods in sequence (not relevant for us, but
this is a killer feature for master-slave systems).
- Each Pod has it's own unique entry in DNS, which makes discovery very
simple (I'll dig into that a bit later)
- Volumes are always mounted to the same Pods, which is very important
in Cache Store scenarios when we restart pods (e.g. Rolling Upgrades [3]).
Thoughts and ideas after spending some time playing with this feature:
- PetSets make discovery a lot easier. It's a combination of two things
- Headless Services [4] which create multiple A records in DNS and
predictable host names. Each Pod has it's own unique DNS entry following
pattern: {PetSetName}-{PodIndex}.{ServiceName} [5]. Here's an example of an
Infinispan PetSet deployed on my local cluster [6]. As you can see we have
all domain names and IPs from a single DNS query.
- Maybe we could perform discovery using this mechanism? I'm aware of
DNS discovery implemented in KUBE_PING [7][8] but the code looks trivial
[9] so maybe it should be implement inside JGroups? @Bela - WDYT?
- PetSets do not integrate well with OpenShift 'new-app' command. In
other words, our users will need to use provided yaml (or json) files to
create Infinispan cluster. It's not a show-stopper but it's a bit less
convenient than 'oc new-app'.
- Since PetSets are alpha resources they need to be considered as
secondary way to deploy Infinispan on Kubernetes and OpenShift.
- Finally, the persistent volumes - since a Pod always gets the same
volume, it would be safe to use any file-based cache store.
If you'd like to play with PetSets on your local environment, here are
necessary yaml files [10].
Thanks
Sebastian
[1] http://kubernetes.io/docs/user-guide/petset/
[2] For checking which APIs are accessible, use 'kubectl api-versions'
[3]
http://infinispan.org/docs/stable/user_guide/user_guide.html#_Rolling_cha...
[4] http://kubernetes.io/docs/user-guide/services/#headless-services
[5] http://kubernetes.io/docs/user-guide/petset/#peer-discovery
[6] https://gist.github.com/slaskawi/0866e63a39276f8ab66376229716a676
[7] https://github.com/jboss-openshift/openshift-ping/tree/master/dns
[8] https://github.com/jgroups-extras/jgroups-kubernetes/tree/master/dns
[9] http://stackoverflow.com/a/12405896/562699
[10] You might need to adjust ImageStream.
https://gist.github.com/slaskawi/7cffb5588dabb770f654557579c5f2d0
8 years, 4 months
Re: [infinispan-dev] Infinispan and OpenShift/Kubernetes PetSets
by Sebastian Laskawiec
Thanks Rob! That clarifies a lot!
On Mon, Aug 22, 2016 at 4:02 PM, Rob Cernich <rcernich(a)redhat.com> wrote:
>
>
> ------------------------------
>
> Hey Rob!
>
> Thanks a lot for clarification!
>
> More comments inlined.
>
> Thanks
> Sebastian
>
> On Sat, Aug 20, 2016 at 12:04 AM, Rob Cernich <rcernich(a)redhat.com> wrote:
>
>> A couple of things...
>>
>> re. volumes:
>> We also need to consider the mounting behavior for scale down scenarios
>> and for overage scenarios when doing upgrades. For the latter, OpenShift
>> can spin up pods of the new version before the older version pods have
>> terminated. This may mean that some volumes from the old pods are
>> orphaned. We did see this when testing A-MQ during upgrades. With a
>> single pod, the upgrade process caused the new version to have a new mount
>> and the original mount was left orphaned (another upgrade would cause the
>> newer pod to pick up the orphaned mount, leaving the new mount orphaned).
>> I believe we worked around this by specifying an overage of 0% during
>> upgrades. This ensured the new pods would pick up the volumes left behind
>> by the old pods. (Actually, we were using subdirectories in the mount,
>> since all pods shared the same volume.)
>>
>>
> I think PetSets try to address this kind of problems. According to the
> manual page [11], the storage is linked to the Pod ordinal and hostaname
> and should be stable.
>
> [11] http://kubernetes.io/docs/user-guide/petset/#when-to-use-pet-set
>
>>
>> re. dns:
>> DNS should work fine as-is, but there are a couple things that you need
>> to consider.
>> 1. Service endpoints are only available in DNS after the pod becomes
>> ready (SVC records on the service name). Because infinispan attaches
>> itself to the cluster, this meant pods were all started as cluster of one,
>> then merged once they noticed the other pods. This had a significant
>> impact on startup. Since then, OpenShift has added the ability to query
>> the endpoints associated with a service as soon as the pod is created,
>> which would allow initialization to work correctly. To make this work,
>> we'd have to change the form of the DNS query to pick up the service
>> endpoints (I forget the naming scheme).
>>
>
> Yes, I agree. Adding nodes one after another will have significant impact
> on cluster startup time. However it should be safe to query the cluster
> (and even put data) during rebalance. So I would say, if a node is up, and
> cluster is not damaged - we should treat it as ready.
>
> To be clear, the issue was around the lifecycle and interaction with the
> readiness probe. OpenShift/Kubernetes only add the pod to the service once
> it's "ready." Our readiness probe defines ready as startup is complete
> (i.e. x of y services started). The issue with this is that as pods come
> up, they only see other pods that are ready when they initialize, so if you
> start with an initial cluster size of five, the new nodes don't see any of
> the other nodes until they finish startup and refresh the list of nodes,
> after which they all have to merge with each other. This had a significant
> impact on performance. Since then, a feature has been added which allows
> you to query DNS for all endpoints regardless of ready state. This allows
> nodes to see the other nodes before they become ready, which allows for a
> more natural cluster formation. To reiterate, the impact on startup was
> significant, especially when scaling up under load.
>
>
> NB - I proposed a HealthCheck API to Infinispan 9 (currently under
> development) [12][13]. The overall cluster health can be in one of 3
> statuses - GREEN (everything is fine), YELLOW (rebalance in progress), RED
> (cluster not healthy). Kubernetes/OpenShift readiness probe should check if
> the status is GREEN or YELLOW. The HealthCheck API is attached to the WF
> management API so you can query it with CURL or using ispn_cli.sh script.
>
> [12] https://github.com/infinispan/infinispan/wiki/Health-check-API
> [13] https://github.com/infinispan/infinispan/pull/4499
>
>>
>> Another thing to keep in mind is that looking up pods by labels allows
>> any pod with the specified label to be added to the cluster. I'm not sure
>> of a use case for this, but it would allow other deployments to be included
>> in the cluster. (You could also argue that the service is the authority
>> for this and any pod with said label would be added as a service endpoint,
>> thus achieving the same behavior...probably more simply too.)
>>
>
> I think this is a scenario when someone might try to attach Infinispan in
> library mode (a dependency in WAR file for example) to the Hot Rod cluster.
> Gustavo answered question like this a while ago [14].
>
> [14] https://developer.jboss.org/message/961568
>
>> Lastly, DNS was a little flaky when we first implemented this, which was
>> part of the reason we went straight to kubernetes. Users were using
>> dnsmasq with wildcards that worked well for routes, but ended up routing
>> services to the router ip instead of pod ip. Needless to say, there were a
>> lot of complications trying to use DNS and debug user problems with service
>> resolution.
>>
>
> I think a governing headless service [15] is required here (PetSets
> require a service but considering how Infinispan works, it should be a
> headless service in my opinion).
>
> [15] http://kubernetes.io/docs/user-guide/services/#headless-services
>
>>
>>
>> Hope that helps,
>> Rob
>>
>> ------------------------------
>>
>> Hey Bela!
>>
>> No no, the resolution can be done with pure JDK.
>>
>> Thanks
>> Sebastian
>>
>> On Fri, Aug 19, 2016 at 11:18 AM, Bela Ban <bban(a)redhat.com> wrote:
>>
>>> Hi Sebastian
>>>
>>> the usual restrictions apply: if DNS discovery depends on external libs,
>>> then it should be hosted in jgroups-extras, otherwise we can add it to
>>> JGroups itself.
>>>
>>> On 19/08/16 11:00, Sebastian Laskawiec wrote:
>>>
>>>> Hey!
>>>>
>>>> I've been playing with Kubernetes PetSets [1] for a while and I'd like
>>>> to share some thoughts. Before I dig in, let me give you some PetSets
>>>> highlights:
>>>>
>>>> * PetSets are alpha resources for managing stateful apps in Kubernetes
>>>> 1.3 (and OpenShift Origin 1.3).
>>>> * Since this is an alpha resource, there are no guarantees about
>>>> backwards compatibility. Alpha resources can also be disabled in
>>>> some public cloud providers (you can control which API versions are
>>>> accessible [2]).
>>>> * PetSets allows starting pods in sequence (not relevant for us, but
>>>> this is a killer feature for master-slave systems).
>>>> * Each Pod has it's own unique entry in DNS, which makes discovery
>>>> very simple (I'll dig into that a bit later)
>>>> * Volumes are always mounted to the same Pods, which is very important
>>>> in Cache Store scenarios when we restart pods (e.g. Rolling Upgrades
>>>> [3]).
>>>>
>>>> Thoughts and ideas after spending some time playing with this feature:
>>>>
>>>> * PetSets make discovery a lot easier. It's a combination of two
>>>> things - Headless Services [4] which create multiple A records in
>>>> DNS and predictable host names. Each Pod has it's own unique DNS
>>>> entry following pattern: {PetSetName}-{PodIndex}.{ServiceName} [5].
>>>> Here's an example of an Infinispan PetSet deployed on my local
>>>> cluster [6]. As you can see we have all domain names and IPs from a
>>>> single DNS query.
>>>> * Maybe we could perform discovery using this mechanism? I'm aware of
>>>> DNS discovery implemented in KUBE_PING [7][8] but the code looks
>>>> trivial [9] so maybe it should be implement inside JGroups? @Bela -
>>>> WDYT?
>>>> * PetSets do not integrate well with OpenShift 'new-app' command. In
>>>> other words, our users will need to use provided yaml (or json)
>>>> files to create Infinispan cluster. It's not a show-stopper but it's
>>>> a bit less convenient than 'oc new-app'.
>>>> * Since PetSets are alpha resources they need to be considered as
>>>> secondary way to deploy Infinispan on Kubernetes and OpenShift.
>>>> * Finally, the persistent volumes - since a Pod always gets the same
>>>> volume, it would be safe to use any file-based cache store.
>>>>
>>>> If you'd like to play with PetSets on your local environment, here are
>>>> necessary yaml files [10].
>>>>
>>>> Thanks
>>>> Sebastian
>>>>
>>>>
>>>> [1] http://kubernetes.io/docs/user-guide/petset/
>>>> [2] For checking which APIs are accessible, use 'kubectl api-versions'
>>>> [3]
>>>> http://infinispan.org/docs/stable/user_guide/user_guide.
>>>> html#_Rolling_chapter
>>>> [4] http://kubernetes.io/docs/user-guide/services/#headless-services
>>>> [5] http://kubernetes.io/docs/user-guide/petset/#peer-discovery
>>>> [6] https://gist.github.com/slaskawi/0866e63a39276f8ab66376229716a676
>>>> [7] https://github.com/jboss-openshift/openshift-ping/tree/master/dns
>>>> [8] https://github.com/jgroups-extras/jgroups-kubernetes/
>>>> tree/master/dns
>>>> [9] http://stackoverflow.com/a/12405896/562699
>>>> [10] You might need to adjust ImageStream.
>>>> https://gist.github.com/slaskawi/7cffb5588dabb770f654557579c5f2d0
>>>>
>>>
>>> --
>>> Bela Ban, JGroups lead (http://www.jgroups.org)
>>>
>>>
>>
>>
>
>
8 years, 4 months
