The Debezium project [1] is working on building change data capture connectors for a variety of databases. MySQL is available now, MongoDB will be soon, and PostgreSQL and Oracle are next on our roadmap. One way in which Debezium and Infinispan can be used together is when Infinispan is being used as a cache for data stored in a database. In this case, Debezium can capture the changes to the database and produce a stream of events; a separate process can consume these change and evict entries from an Infinispan cache. If Infinispan is to be used as a data store, then it would be useful for Debezium to be able to capture those changes so other apps/services can consume the changes. First of all, does this make sense? Secondly, if it does, then Debezium would need an Infinispan connector, and it’s not clear to me how that connector might capture the changes from Infinispan. Debezium typically monitors the log of transactions/changes that are committed to a database. Of course how this works varies for each type of database. For example, MySQL internally produces a transaction log that contains information about every committed row change, and MySQL ensures that every committed change is included and that non-committed changes are excluded. The MySQL mechanism is actually part of the replication mechanism, so slaves update their internal state by reading the master’s log. The Debezium MySQL connector [2] simply reads the same log. Infinispan has several mechanisms that may be useful: * Interceptors - See [3]. This seems pretty straightforward and IIUC provides access to all internal operations. However, it’s not clear to me whether a single interceptor will see all the changes in a cluster (perhaps in local and replicated modes) or only those changes that happen on that particular node (in distributed mode). It’s also not clear whether this interceptor is called within the context of the cache’s transaction, so if a failure happens just at the wrong time whether a change might be made to the cache but is not seen by the interceptor (or vice versa). * Cross-site replication - See [4][5]. A potential advantage of this mechanism appears to be that it is defined (more) globally, and it appears to function if the remote backup comes back online after being offline for a period of time. * State transfer - is it possible to participate as a non-active member of the cluster, and to effectively read all state transfer activities that occur within the cluster? * Cache store - tie into the cache store mechanism, perhaps by wrapping an existing cache store and sitting between the cache and the cache store * Monitor the cache store - don’t monitor Infinispan at all, and instead monitor the store in which Infinispan is storing entries. (This is probably the least attractive, since some stores can’t be monitored, or because the store is persisting an opaque binary value.) Are there other mechanism that might be used? There are a couple of important requirements for change data capture to be able to work correctly: 1. Upon initial connection, the CDC connector must be able to obtain a snapshot of all existing data, followed by seeing all changes to data that may have occurred since the snapshot was started. If the connector is stopped/fails, upon restart it needs to be able to reconnect and either see all changes that occurred since it last was capturing changes, or perform a snapshot. (Performing a snapshot upon restart is very inefficient and undesirable.) This works as follows: the CDC connector only records the “offset” in the source’s sequence of events; what this “offset” entails depends on the source. Upon restart, the connector can use this offset information to coordinate with the source where it wants to start reading. (In MySQL and PostgreSQL, every event includes the filename of the log and position in that file. MongoDB includes in each event the monotonically increasing timestamp of the transaction. 2. No change can be missed, even when things go wrong and components crash. 3. When a new entry is added, the “after” state of the entity will be included. When an entry is updated, the “after” state will be included in the event; if possible, the event should also include the “before” state. When an entry is removed, the “before” state should be included in the event. Any thoughts or advice would be greatly appreciated. Best regards, Randall [1] http://debezium.io [2] http://debezium.io/docs/connectors/mysql/ [3] http://infinispan.org/docs/stable/user_guide/user_guide.html#_custom_inte... [4] http://infinispan.org/docs/stable/user_guide/user_guide.html#CrossSiteRep... [5] https://github.com/infinispan/infinispan/wiki/Design-For-Cross-Site-Repli... _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

On 11 Jul 2016, at 16:41, Randall Hauch <rhauch(a)redhat.com&gt; wrote: > > On Jul 11, 2016, at 3:42 AM, Adrian Nistor <anistor(a)redhat.com&gt; wrote: > > Hi Randall, > > Infinispan supports both push and pull access models. The push model is supported by events (and listeners), which are cluster wide and are available in both library and remote mode (hotrod). The notification system is pretty advanced as there is a filtering mechanism available that can use a hand coded filter / converter or one specified in jpql (experimental atm). Getting a snapshot of the initial data is also possible. But infinispan does not produce a transaction log to be used for determining all changes that happened since a previous connection time, so you'll always have to get a new full snapshot when re-connecting. > > So if Infinispan is the data store I would base the Debezium connector implementation on Infinispan's event notification system. Not sure about the other use case though. > Thanks, Adrian, for the feedback. A couple of questions. You mentioned Infinispan has a pull model — is this just using the normal API to read the entries? With event listeners, a single connection will receive all of the events that occur in the cluster, correct? Is it possible (e.g., a very unfortunately timed crash) for a change to be made to the cache without an event being produced and sent to listeners?

What happens if the network fails or partitions? How does cross site replication address this?

Has there been any thought about adding to Infinispan a write ahead log or transaction log to each node or, better yet, for the whole cluster?

Thanks again! > Adrian > > On 07/09/2016 04:38 PM, Randall Hauch wrote: >> The Debezium project [1] is working on building change data capture connectors for a variety of databases. MySQL is available now, MongoDB will be soon, and PostgreSQL and Oracle are next on our roadmap. >> >> One way in which Debezium and Infinispan can be used together is when Infinispan is being used as a cache for data stored in a database. In this case, Debezium can capture the changes to the database and produce a stream of events; a separate process can consume these change and evict entries from an Infinispan cache. >> >> If Infinispan is to be used as a data store, then it would be useful for Debezium to be able to capture those changes so other apps/services can consume the changes. First of all, does this make sense? Secondly, if it does, then Debezium would need an Infinispan connector, and it’s not clear to me how that connector might capture the changes from Infinispan. >> >> Debezium typically monitors the log of transactions/changes that are committed to a database. Of course how this works varies for each type of database. For example, MySQL internally produces a transaction log that contains information about every committed row change, and MySQL ensures that every committed change is included and that non-committed changes are excluded. The MySQL mechanism is actually part of the replication mechanism, so slaves update their internal state by reading the master’s log. The Debezium MySQL connector [2] simply reads the same log. >> >> Infinispan has several mechanisms that may be useful: >> >> • Interceptors - See [3]. This seems pretty straightforward and IIUC provides access to all internal operations. However, it’s not clear to me whether a single interceptor will see all the changes in a cluster (perhaps in local and replicated modes) or only those changes that happen on that particular node (in distributed mode). It’s also not clear whether this interceptor is called within the context of the cache’s transaction, so if a failure happens just at the wrong time whether a change might be made to the cache but is not seen by the interceptor (or vice versa). >> • Cross-site replication - See [4][5]. A potential advantage of this mechanism appears to be that it is defined (more) globally, and it appears to function if the remote backup comes back online after being offline for a period of time. >> • State transfer - is it possible to participate as a non-active member of the cluster, and to effectively read all state transfer activities that occur within the cluster? >> • Cache store - tie into the cache store mechanism, perhaps by wrapping an existing cache store and sitting between the cache and the cache store >> • Monitor the cache store - don’t monitor Infinispan at all, and instead monitor the store in which Infinispan is storing entries. (This is probably the least attractive, since some stores can’t be monitored, or because the store is persisting an opaque binary value.) >> >> Are there other mechanism that might be used? >> >> There are a couple of important requirements for change data capture to be able to work correctly: >> >> • Upon initial connection, the CDC connector must be able to obtain a snapshot of all existing data, followed by seeing all changes to data that may have occurred since the snapshot was started. If the connector is stopped/fails, upon restart it needs to be able to reconnect and either see all changes that occurred since it last was capturing changes, or perform a snapshot. (Performing a snapshot upon restart is very inefficient and undesirable.) This works as follows: the CDC connector only records the “offset” in the source’s sequence of events; what this “offset” entails depends on the source. Upon restart, the connector can use this offset information to coordinate with the source where it wants to start reading. (In MySQL and PostgreSQL, every event includes the filename of the log and position in that file. MongoDB includes in each event the monotonically increasing timestamp of the transaction. >> • No change can be missed, even when things go wrong and components crash. >> • When a new entry is added, the “after” state of the entity will be included. When an entry is updated, the “after” state will be included in the event; if possible, the event should also include the “before” state. When an entry is removed, the “before” state should be included in the event. >> >> Any thoughts or advice would be greatly appreciated. >> >> Best regards, >> >> Randall >> >> >> [1] http://debezium.io >> [2] http://debezium.io/docs/connectors/mysql/ >> [3] http://infinispan.org/docs/stable/user_guide/user_guide.html#_custom_inte... >> [4] http://infinispan.org/docs/stable/user_guide/user_guide.html#CrossSiteRep... >> [5] https://github.com/infinispan/infinispan/wiki/Design-For-Cross-Site-Repli... >> >> >> _______________________________________________ >> infinispan-dev mailing list >> >> infinispan-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev > _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

Reviving this old thread, and as before I appreciate any help the Infinispan community might provide. There definitely is interest in Debezium capturing the changes being made to an Infinispan cluster. This isn’t as important when Infinispan is used as a cache, but when Infinispan is used as a store then it is important for other apps/services to be able to accurately keep up with the changes being made in the store. On Jul 29, 2016, at 8:47 AM, Galder Zamarreño <galder(a)redhat.com&gt; wrote: -- Galder Zamarreño Infinispan, Red Hat On 11 Jul 2016, at 16:41, Randall Hauch <rhauch(a)redhat.com&gt; wrote: On Jul 11, 2016, at 3:42 AM, Adrian Nistor <anistor(a)redhat.com&gt; wrote: Hi Randall, Infinispan supports both push and pull access models. The push model is supported by events (and listeners), which are cluster wide and are available in both library and remote mode (hotrod). The notification system is pretty advanced as there is a filtering mechanism available that can use a hand coded filter / converter or one specified in jpql (experimental atm). Getting a snapshot of the initial data is also possible. But infinispan does not produce a transaction log to be used for determining all changes that happened since a previous connection time, so you'll always have to get a new full snapshot when re-connecting. So if Infinispan is the data store I would base the Debezium connector implementation on Infinispan's event notification system. Not sure about the other use case though. Thanks, Adrian, for the feedback. A couple of questions. You mentioned Infinispan has a pull model — is this just using the normal API to read the entries? With event listeners, a single connection will receive all of the events that occur in the cluster, correct? Is it possible (e.g., a very unfortunately timed crash) for a change to be made to the cache without an event being produced and sent to listeners? ^ Yeah, that can happen due to async nature of remote events. However, there's the possibility for clients, upon receiving a new topology, to receive the current state of the server as events, see [1] and [2] [1] http://infinispan.org/docs/dev/user_guide/user_ guide.html#client_event_listener_state_consumption [2] http://infinispan.org/docs/dev/user_guide/user_ guide.html#client_event_listener_failure_handling It is critical that any change event stream is consistent with the store, and the change event stream is worthless without it. Only when the change event stream is an accurate representation of what changed can downstream consumers use the stream to rebuild their own perfect copy of the upstream store and to keep those copies consistent with the upstream store. So, given that the events are handled asynchronously, in a cluster how are multiple changes to a single entry handled. For example, if a client sets entry <A,Foo>, then a short time after that (or another) client sets entry <A,Bar>, is it guaranteed that a client listening to events will see <A,Foo> first and <A,Bar> some time later? Or is it possible that a client listening might first see <A,Bar> and then <A,Foo>? What happens if the network fails or partitions? How does cross site replication address this? In terms of cross-site, depends what the client is connected to. Clients can now failover between sites, so they should be able to deal with events too in the same as explained above. Has there been any thought about adding to Infinispan a write ahead log or transaction log to each node or, better yet, for the whole cluster? Not that I'm aware of but we've recently added security audit log, so a transaction log might make sense too. Without a transaction log, Debezium would have to use a client listener with includeCurrentState=true to obtain the state every time it reconnects. If Debezium just included all of this state in the event stream, then the stream might contain lots of superfluous or unnecessary events, then this impacts all downstream consumers by forcing them to spend a lot of time processing changes that never really happened. So the only way to avoid that would be for Debezium to use an external store to track the changes it has seen so far so that it doesn’t include unnecessary events in the change event stream. It’d be a shame to have to require this much infrastructure. A transaction log would really be a great way to solve this problem. Has there been any more thought about Infinispan using and exposing a transaction log? Or perhaps Infinispan could record the changes in a Kafka topic directly? (I guess if the Infinispan cache used relational database(s) as a cache store(s), then Debezium could just capture the changes from there. That seems like a big constraint, though.) Thoughts?

I recently updated a proposal [1] based on several discussions we had in the past that is essentially about introducing an event storage mechanism (write ahead log) in order to improve reliability, failover and "replayability" for the remote listeners, any feedback greatly appreciated.

[1] https://github.com/infinispan/infinispan/wiki/Remote-Listeners-improvemen... Thanks, Gustavo _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

> On Dec 9, 2016, at 3:13 AM, Radim Vansa <rvansa(a)redhat.com&gt; wrote: > > On 12/08/2016 10:13 AM, Gustavo Fernandes wrote: >> >> I recently updated a proposal [1] based on several discussions we had >> in the past that is essentially about introducing an event storage >> mechanism (write ahead log) in order to improve reliability, failover >> and "replayability" for the remote listeners, any feedback greatly >> appreciated. > > Hi Gustavo, > > while I really like the pull-style architecture and reliable events, I > see some problematic parts here: > > 1) 'cache that would persist the events with a monotonically increasing id' > > I assume that you mean globally (for all entries) monotonous. How will > you obtain such ID? Currently, commands have unique IDs that are > <Address, Long> where the number part is monotonous per node. That's > easy to achieve. But introducing globally monotonous counter means that > there will be a single contention point. (you can introduce another > contention points by adding backups, but this is probably unnecessary as > you can find out the last id from the indexed cache data). Per-segment > monotonous would be probably more scalabe, though that increases complexity. It is complicated, but one way to do this is to have one “primary” node maintain the log and to have other replicate from it. The cluster does need to use consensus to agree which is the primary, and to know which secondary becomes the primary if the primary is failing. Consensus is not trivial, but JGroups Raft (http://belaban.github.io/jgroups-raft/ <http://belaban.github.io/jgroups-raft/>) may be an option. However, this approach ensures that the replica logs are identical to the primary since they are simply recording the primary’s log as-is. Of course, another challenge is what happens during a failure of the primary log node, and can any transactions be performed/completed while the primary is unavailable. Another option is to have each node maintain their own log, and to have an aggregator log that merges/combines the various logs into one. Not sure how feasible it is to merge logs by getting rid of duplicates and determining a total order, but if it is then it may have better fault tolerance characteristics. Of course, it is possible to have node-specific monotonic IDs. For example, MySQL Global Transaction IDs (GTIDs) use a unique UUID for each node, and then GTIDs consists of the node’s UUID plus a monotonically-increasing value (e.g., “31fc48cd-ecd4-46ad-b0a9-f515fc9497c4:1001”). The transaction log contains a mix of GTIDs, and MySQL replication uses a “GTID set” to describe the ranges of transactions known by a server (e.g., “u1:1-100,u2:1-10000,u3:3-5” where “u1”, “u2”, and “u3” are actually UUIDs). So, when a MySQL replica connects, it says “I know about this GTID set", and this tells the master where that client wants to start reading. > > 2) 'The write to the event log would be async in order to not affect > normal data writes' > > Who should write to the cache? > a) originator - what if originator crashes (despite the change has been > added)? Besides, originator would have to do (async) RPC to primary > owner (which will be the primary owner of the event, too). > b) primary owner - with triangle, primary does not really know if the > change has been written on backup. Piggybacking that info won't be > trivial - we don't want to send another message explicitly. But even if > we get the confirmation, since the write to event cache is async, if the > primary owner crashes before replicating the event to backup, we lost > the event > c) all owners, but locally - that will require more complex > reconciliation if the event did really happen on all surviving nodes or > not. And backups could have some trouble to resolve order, too. > > IIUC clustered listeners are called from primary owner before the change > is really confirmed on backups (@Pedro correct me if I am wrong, > please), but for this reliable event cache you need higher level of > consistency. This could be handled by writing a confirmation or “commit” event to the log when the write is confirmed or the transaction is committed. Then, only those confirmed events/transactions would be exposed to client listeners. This requires some buffering, but this could be done in each HotRod client. > > 3) The log will also have to filter out retried operations (based on > command ID - though this can be indexed, too). Though, I would prefer to > see per-event command-id log to deal with retries properly. IIUC, a “commit” event would work here, too. > > 4) Client should pull data, but I would keep push notifications that > 'something happened' (throttled on server). There could be use case for > rarely updated caches, and polling the servers would be excessive there. IMO the clients should poll, but if the server has nothing to return it blocks until there is something or until a timeout occurs. This makes it easy for clients and actually reduces network traffic compared to constantly polling. BTW, a lot of this is replicating the functionality of Kafka, which is already quite mature and feature rich. It’s actually possible to *embed* Kafka to simplify operations, but I don’t think that’s recommended. And, it introduces a very complex codebase that would need to be supported. > > Radim > >> >> >> [1] >> https://github.com/infinispan/infinispan/wiki/Remote-Listeners-improvemen... >> >> Thanks, >> Gustavo >> >> >> >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev > > > -- > Radim Vansa <rvansa(a)redhat.com&gt; > JBoss Performance Team > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev

_______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

On 12/09/2016 06:25 PM, Emmanuel Bernard wrote: > Randall and I had a chat on $subject. Here is a proposal worth > exploring as it is very lightweight on Infinispan's code. > > Does an operation has a unique id sahred by the master and replicas? > If not could we add that? Yes, each modification has unique CommandInvocationId in non-tx cache, and there are GlobalTransaction ids in tx-caches. > > The proposal itself: > > The total order would not be global but per key. > Each node has a Debezium connector instance embedded that listens to the > operations happening (primary and replicas alike). > All of this process is happening async compared to the operation. > Per key, a log of operations is kept in memory (it contains the key, the > operation, the operation unique id and a ack status. > If on the key owner, the operation is written by the Debezium connector > to Kafka when it has been acked (whatever that means is where I'm less > knowledgable - too many bi-cache, tri-cache and quadri latency mixed in > my brain). And the ack is what you have to define. All Infinispan gives you is operation was confirmed on originator => all owners (primary + backups) have stored the value If you send the ack from originator to primary, it could be lost (when originator crashes). If you write Kafka on originator, you don't have any order, and the update could be lost by crashing before somehow replicating to Kafka. If you write Kafka on primary, you need the ack from all backups (minor technical difficulty), and if primary crashes after it has sent the update to all backups, data is effectively modified but Kafka is not. The originator has to detect primary crashing to retry - so probably the primary could only send the ack to originator after it gets ack from all backups AND updates Kafka. But this is exactly what Triangle eliminated. And you still have the problem when originator crashes as well, but at least you're resilient to single node (primary) failure. So you probably intend to forget any "acks" and as soon as primary executes the write locally, just push it to Kafka. No matter the actual "outcome" of the operation. E.g. with putIfAbsent there could be topology change during replication to backup, which will cause the operation to be retried (from the originator). In the meantime, there would be another write, and the retried putIfAbsent will fail. You will have one successful and one unsuccessful putIfAbsent in the log, with the same ID. > On a replica, the kafka partition is read regularly to clear the > in-memory log from operations stored in Kafka > If the replica becomes the owner, it reads the kafka partition to see > what operations are already in and writes the missing ones. Backup owner (becoming primary owner) having the write locally logged does not mean that operation was successfully finished on all owners. > > There are a few cool things: > - few to no change in what Infinispan does > - no global ordering simplifies things and frankly is fine for most > Debezium cases. In the end a global order could be defined after the > fact (by not partitioning for example). But that's a pure downstream > concern. > - everything is async compared to the Infinispan ops > - the in-memory log can remain in memory as it is protected by replicas > - the in-memory log is self cleaning thanks to the state in Kafka > > Everyone wins. But it does require some sort of globally unique id per > operation to dedup. And a suitable definition for Debezium if the operation "happened" or not. Radim > > Emmanuel > > > On Fri 16-12-09 10:08, Randall Hauch wrote: >>> On Dec 9, 2016, at 3:13 AM, Radim Vansa <rvansa(a)redhat.com&gt; wrote: >>> >>> On 12/08/2016 10:13 AM, Gustavo Fernandes wrote: >>>> I recently updated a proposal [1] based on several discussions we had >>>> in the past that is essentially about introducing an event storage >>>> mechanism (write ahead log) in order to improve reliability, failover >>>> and "replayability" for the remote listeners, any feedback greatly >>>> appreciated. >>> Hi Gustavo, >>> >>> while I really like the pull-style architecture and reliable events, I >>> see some problematic parts here: >>> >>> 1) 'cache that would persist the events with a monotonically increasing id' >>> >>> I assume that you mean globally (for all entries) monotonous. How will >>> you obtain such ID? Currently, commands have unique IDs that are >>> <Address, Long> where the number part is monotonous per node. That's >>> easy to achieve. But introducing globally monotonous counter means that >>> there will be a single contention point. (you can introduce another >>> contention points by adding backups, but this is probably unnecessary as >>> you can find out the last id from the indexed cache data). Per-segment >>> monotonous would be probably more scalabe, though that increases complexity. >> It is complicated, but one way to do this is to have one “primary” node maintain the log and to have other replicate from it. The cluster does need to use consensus to agree which is the primary, and to know which secondary becomes the primary if the primary is failing. Consensus is not trivial, but JGroups Raft (http://belaban.github.io/jgroups-raft/ <http://belaban.github.io/jgroups-raft/>) may be an option. However, this approach ensures that the replica logs are identical to the primary since they are simply recording the primary’s log as-is. Of course, another challenge is what happens during a failure of the primary log node, and can any transactions be performed/completed while the primary is unavailable. >> >> Another option is to have each node maintain their own log, and to have an aggregator log that merges/combines the various logs into one. Not sure how feasible it is to merge logs by getting rid of duplicates and determining a total order, but if it is then it may have better fault tolerance characteristics. >> >> Of course, it is possible to have node-specific monotonic IDs. For example, MySQL Global Transaction IDs (GTIDs) use a unique UUID for each node, and then GTIDs consists of the node’s UUID plus a monotonically-increasing value (e.g., “31fc48cd-ecd4-46ad-b0a9-f515fc9497c4:1001”). The transaction log contains a mix of GTIDs, and MySQL replication uses a “GTID set” to describe the ranges of transactions known by a server (e.g., “u1:1-100,u2:1-10000,u3:3-5” where “u1”, “u2”, and “u3” are actually UUIDs). So, when a MySQL replica connects, it says “I know about this GTID set", and this tells the master where that client wants to start reading. >> >>> 2) 'The write to the event log would be async in order to not affect >>> normal data writes' >>> >>> Who should write to the cache? >>> a) originator - what if originator crashes (despite the change has been >>> added)? Besides, originator would have to do (async) RPC to primary >>> owner (which will be the primary owner of the event, too). >>> b) primary owner - with triangle, primary does not really know if the >>> change has been written on backup. Piggybacking that info won't be >>> trivial - we don't want to send another message explicitly. But even if >>> we get the confirmation, since the write to event cache is async, if the >>> primary owner crashes before replicating the event to backup, we lost >>> the event >>> c) all owners, but locally - that will require more complex >>> reconciliation if the event did really happen on all surviving nodes or >>> not. And backups could have some trouble to resolve order, too. >>> >>> IIUC clustered listeners are called from primary owner before the change >>> is really confirmed on backups (@Pedro correct me if I am wrong, >>> please), but for this reliable event cache you need higher level of >>> consistency. >> This could be handled by writing a confirmation or “commit” event to the log when the write is confirmed or the transaction is committed. Then, only those confirmed events/transactions would be exposed to client listeners. This requires some buffering, but this could be done in each HotRod client. >> >>> 3) The log will also have to filter out retried operations (based on >>> command ID - though this can be indexed, too). Though, I would prefer to >>> see per-event command-id log to deal with retries properly. >> IIUC, a “commit” event would work here, too. >> >>> 4) Client should pull data, but I would keep push notifications that >>> 'something happened' (throttled on server). There could be use case for >>> rarely updated caches, and polling the servers would be excessive there. >> IMO the clients should poll, but if the server has nothing to return it blocks until there is something or until a timeout occurs. This makes it easy for clients and actually reduces network traffic compared to constantly polling. >> >> BTW, a lot of this is replicating the functionality of Kafka, which is already quite mature and feature rich. It’s actually possible to *embed* Kafka to simplify operations, but I don’t think that’s recommended. And, it introduces a very complex codebase that would need to be supported. >> >>> Radim >>> >>>> >>>> [1] >>>> https://github.com/infinispan/infinispan/wiki/Remote-Listeners-improvemen... >>>> >>>> Thanks, >>>> Gustavo >>>> >>>> >>>> >>>> _______________________________________________ >>>> infinispan-dev mailing list >>>> infinispan-dev(a)lists.jboss.org >>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>> >>> -- >>> Radim Vansa <rvansa(a)redhat.com&gt; >>> JBoss Performance Team >>> >>> _______________________________________________ >>> infinispan-dev mailing list >>> infinispan-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev

On 16 Dec 2016, at 08:46, Tristan Tarrant <ttarrant(a)redhat.com&gt; wrote: On 09/12/16 18:25, Emmanuel Bernard wrote: > The total order would not be global but per key. > Each node has a Debezium connector instance embedded that listens to the > operations happening (primary and replicas alike). > All of this process is happening async compared to the operation. > Per key, a log of operations is kept in memory (it contains the key, the > operation, the operation unique id and a ack status. > If on the key owner, the operation is written by the Debezium connector > to Kafka when it has been acked (whatever that means is where I'm less > knowledgable - too many bi-cache, tri-cache and quadri latency mixed in > my brain). > On a replica, the kafka partition is read regularly to clear the > in-memory log from operations stored in Kafka > If the replica becomes the owner, it reads the kafka partition to see Yes, the above design is what sprung to mind initially. Not sure about the need of keeping the log in memory, as we would probably need some form of persistent log for cache shutdown. Since this looks a lot like the append-log of the Artemis journal, maybe we could use that.

One thing I missed, is whether Debezium is interested in transaction boundaries.

On 16 Dec 2016, at 09:48, Tristan Tarrant <ttarrant(a)redhat.com&gt; wrote: On 16/12/16 09:34, Emmanuel Bernard wrote: >> Yes, the above design is what sprung to mind initially. Not sure about >> the need of keeping the log in memory, as we would probably need some >> form of persistent log for cache shutdown. Since this looks a lot like >> the append-log of the Artemis journal, maybe we could use that. > > Well, when the cache is shut down, don’t we have time to empty the in-memory log? Cache shutdown should not be deferred because there is a backlog of events that haven't been forwarded to Debezium, so we would want to pick up from where we were when we restart the cache.

> On 16 Dec 2016, at 09:48, Tristan Tarrant <ttarrant(a)redhat.com&gt; wrote: > > On 16/12/16 09:34, Emmanuel Bernard wrote: >>> Yes, the above design is what sprung to mind initially. Not sure about >>> the need of keeping the log in memory, as we would probably need some >>> form of persistent log for cache shutdown. Since this looks a lot like >>> the append-log of the Artemis journal, maybe we could use that. >> >> Well, when the cache is shut down, don’t we have time to empty the

> > Cache shutdown should not be deferred because there is a backlog of > events that haven't been forwarded to Debezium, so we would want to pick > up from where we were when we restart the cache. But you’re willing to wait for the Artemis journal finish writing? I

On 16 Dec 2016, at 16:25, Emmanuel Bernard <emmanuel(a)hibernate.org&gt; wrote: > On 16 Dec 2016, at 13:38, Tristan Tarrant <ttarrant(a)redhat.com&gt; wrote: > > On 16/12/16 13:12, Emmanuel Bernard wrote: >> >>> On 16 Dec 2016, at 09:48, Tristan Tarrant <ttarrant(a)redhat.com&gt; wrote: >>> >>> On 16/12/16 09:34, Emmanuel Bernard wrote: >>>>> Yes, the above design is what sprung to mind initially. Not sure about >>>>> the need of keeping the log in memory, as we would probably need some >>>>> form of persistent log for cache shutdown. Since this looks a lot like >>>>> the append-log of the Artemis journal, maybe we could use that. >>>> >>>> Well, when the cache is shut down, don’t we have time to empty the in-memory log? >>> >>> Cache shutdown should not be deferred because there is a backlog of >>> events that haven't been forwarded to Debezium, so we would want to pick >>> up from where we were when we restart the cache. >> >> But you’re willing to wait for the Artemis journal finish writing? I don’t quite see the difference. > > I'm thinking about the case where Debezium is temporarily not able to > collect the changes. I’m thinking about the case where Artemis is not able to collect the changes. How is that different? :)

On 16 Dec 2016, at 17:00, Tristan Tarrant <ttarrant(a)redhat.com&gt; wrote: On 16/12/16 16:30, Emmanuel Bernard wrote: > The Emmanuel Dec 9th proposal handles I think the case of topology changes and nodes going down. We need a better name :) > For a cluster-wide shutdown with no time to flush queues, I think both Artemis and the local debezium talking to remote Kafka will be in trouble. > > The main difference is I imagine that your Artemis log will be local to the node being shut down. But that would mean a complex system to restart with the unflushed queue from all node that were shut down. Are we there yet? I'm specifically thinking about when the Debezium embedded in Infinispan cannot talk to Kafka for whatever reason, and the user wants to shutdown. Tristan -- Tristan Tarrant Infinispan Lead JBoss, a division of Red Hat _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

> On Dec 9, 2016, at 3:13 AM, Radim Vansa <rvansa(a)redhat.com > <mailto:rvansa@redhat.com>> wrote: > > On 12/08/2016 10:13 AM, Gustavo Fernandes wrote: >> >> I recently updated a proposal [1] based on several discussions we had >> in the past that is essentially about introducing an event storage >> mechanism (write ahead log) in order to improve reliability, failover >> and "replayability" for the remote listeners, any feedback greatly >> appreciated. > > Hi Gustavo, > > while I really like the pull-style architecture and reliable events, I > see some problematic parts here: > > 1) 'cache that would persist the events with a monotonically > increasing id' > > I assume that you mean globally (for all entries) monotonous. How will > you obtain such ID? Currently, commands have unique IDs that are > <Address, Long> where the number part is monotonous per node. That's > easy to achieve. But introducing globally monotonous counter means that > there will be a single contention point. (you can introduce another > contention points by adding backups, but this is probably unnecessary as > you can find out the last id from the indexed cache data). Per-segment > monotonous would be probably more scalabe, though that increases > complexity. It is complicated, but one way to do this is to have one “primary” node maintain the log and to have other replicate from it. The cluster does need to use consensus to agree which is the primary, and to know which secondary becomes the primary if the primary is failing. Consensus is not trivial, but JGroups Raft (http://belaban.github.io/jgroups-raft/) may be an option. However, this approach ensures that the replica logs are identical to the primary since they are simply recording the primary’s log as-is. Of course, another challenge is what happens during a failure of the primary log node, and can any transactions be performed/completed while the primary is unavailable.

Another option is to have each node maintain their own log, and to have an aggregator log that merges/combines the various logs into one. Not sure how feasible it is to merge logs by getting rid of duplicates and determining a total order, but if it is then it may have better fault tolerance characteristics. Of course, it is possible to have node-specific monotonic IDs. For example, MySQL Global Transaction IDs (GTIDs) use a unique UUID for each node, and then GTIDs consists of the node’s UUID plus a monotonically-increasing value (e.g., “31fc48cd-ecd4-46ad-b0a9-f515fc9497c4:1001”). The transaction log contains a mix of GTIDs, and MySQL replication uses a “GTID set” to describe the ranges of transactions known by a server (e.g., “u1:1-100,u2:1-10000,u3:3-5” where “u1”, “u2”, and “u3” are actually UUIDs). So, when a MySQL replica connects, it says “I know about this GTID set", and this tells the master where that client wants to start reading.

> > 2) 'The write to the event log would be async in order to not affect > normal data writes' > > Who should write to the cache? > a) originator - what if originator crashes (despite the change has been > added)? Besides, originator would have to do (async) RPC to primary > owner (which will be the primary owner of the event, too). > b) primary owner - with triangle, primary does not really know if the > change has been written on backup. Piggybacking that info won't be > trivial - we don't want to send another message explicitly. But even if > we get the confirmation, since the write to event cache is async, if the > primary owner crashes before replicating the event to backup, we lost > the event > c) all owners, but locally - that will require more complex > reconciliation if the event did really happen on all surviving nodes or > not. And backups could have some trouble to resolve order, too. > > IIUC clustered listeners are called from primary owner before the change > is really confirmed on backups (@Pedro correct me if I am wrong, > please), but for this reliable event cache you need higher level of > consistency. This could be handled by writing a confirmation or “commit” event to the log when the write is confirmed or the transaction is committed. Then, only those confirmed events/transactions would be exposed to client listeners. This requires some buffering, but this could be done in each HotRod client.

> > 3) The log will also have to filter out retried operations (based on > command ID - though this can be indexed, too). Though, I would prefer to > see per-event command-id log to deal with retries properly. IIUC, a “commit” event would work here, too. > > 4) Client should pull data, but I would keep push notifications that > 'something happened' (throttled on server). There could be use case for > rarely updated caches, and polling the servers would be excessive there. IMO the clients should poll, but if the server has nothing to return it blocks until there is something or until a timeout occurs. This makes it easy for clients and actually reduces network traffic compared to constantly polling.

BTW, a lot of this is replicating the functionality of Kafka, which is already quite mature and feature rich. It’s actually possible to *embed* Kafka to simplify operations, but I don’t think that’s recommended. And, it introduces a very complex codebase that would need to be supported.

> > Radim > >> >> >> [1] >> https://github.com/infinispan/infinispan/wiki/Remote-Listeners-improvemen... >> >> Thanks, >> Gustavo >> >> >> >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev > > > -- > Radim Vansa <rvansa(a)redhat.com <mailto:rvansa@redhat.com>> > JBoss Performance Team > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org <mailto:infinispan-dev@lists.jboss.org> > https://lists.jboss.org/mailman/listinfo/infinispan-dev _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

But introducing globally monotonous counter means that there will be a single contention point.

In short, what's the ultimate goal? I see two main but different options intertwined: - allow to synchronize the *final state* of a replica

- inspect specific changes For the first case, it would be enough for us to be able to provide a "squashed history" (as in Git squash), but we'd need to keep versioned shapshots around and someone needs to tell you which ones can be garbage collected. For example when a key is: written, updated, updated, deleted since the snapshot, we'll send only "deleted" as the intermediary states are irrelevant. For the second case, say the goal is to inspect fluctuations of price variations of some item, then the intermediary states are not irrelevant. Which one will we want to solve? Both?

Personally the attempt of solving the second one seems like a huge pivot of the project, the current data-structures and storage are not designed for this.

I see the value of such benefits, but maybe Infinispan is not the right tool for such a problem. I'd prefer to focus on the benefits of the squashed history, and have versioned entries soon, but even in that case we need to define which versions need to be kept around, and how garbage collection / vacuuming is handled.

In short, I'd like to see an agreement that analyzing e.g. fluctuations in stock prices would be a non-goal, if these are stored as {"stock name", value} key/value pairs. One could still implement such a thing by using a more sophisticated model, just don't expect to be able to see all intermediary values each entry has ever had since the key was first used.

On Mon, Dec 12, 2016 at 3:13 PM, Sanne Grinovero <sanne(a)infinispan.org&gt; wrote: > > In short, what's the ultimate goal? I see two main but different > options intertwined: > - allow to synchronize the *final state* of a replica I'm assuming this case is already in place when using remote listeners and includeCurrentState=true and we are discussing how to improve it, as described in the proposal in the wiki and on the 5th email of this thread. > > - inspect specific changes > > For the first case, it would be enough for us to be able to provide a > "squashed history" (as in Git squash), but we'd need to keep versioned > shapshots around and someone needs to tell you which ones can be > garbage collected. > For example when a key is: written, updated, updated, deleted since > the snapshot, we'll send only "deleted" as the intermediary states are > irrelevant. > For the second case, say the goal is to inspect fluctuations of price > variations of some item, then the intermediary states are not > irrelevant. > > Which one will we want to solve? Both? Looking at http://debezium.io/, it implies the second case.

"[...] Start it up, point it at your databases, and your apps can start responding to all of the inserts, updates, and deletes that other apps commit to your databases. [...] your apps can respond quickly and never miss an event, even when things go wrong." IMO the choice between squashed/full history, and even retention time is highly application specific. Deletes might not even be involved, one may be interested on answering "what is the peak value of a certain key during the day?"

> Personally the attempt of solving the second one seems like a huge > pivot of the project, the current data-structures and storage are not > designed for this. +1, as I wrote earlier about ditching the idea of event cache storage in favor of Lucene.

> I see the value of such benefits, but maybe > Infinispan is not the right tool for such a problem. > > I'd prefer to focus on the benefits of the squashed history, and have > versioned entries soon, but even in that case we need to define which > versions need to be kept around, and how garbage collection / > vacuuming is handled. Is that proposal written/recorded somewhere? It'd be interesting to know how a client interested on data changes would consume those multi-versioned entries (push/pull with offset?, sorted/unsorted?, client tracking/per key/per version?), as it seems there is some storage impedance as well. > > > In short, I'd like to see an agreement that analyzing e.g. > fluctuations in stock prices would be a non-goal, if these are stored > as {"stock name", value} key/value pairs. One could still implement > such a thing by using a more sophisticated model, just don't expect to > be able to see all intermediary values each entry has ever had since > the key was first used. Continuous Queries listens to data key/value data using a query, should it not be expected to see all the intermediary values when changes in the server causes an entry to start/stop matching the query?

On 12 Dec 2016, at 16:13, Sanne Grinovero <sanne(a)infinispan.org&gt; wrote: I'm reading many clever suggestions for various aspects of such a system, but I fail to see a clear definition of the goal. > From Randall's opening email I understand how MySQL does this, but it's an example and I'm not sure which aspects are implementation details of how MySQL happens to accomplish this, and which aspects are requirements for the Infinispan enhancement proposals. I remember a meeting with Manik Surtani, Jonathan Halliday and Mark Little, whose outcome was a general agreement that Infinispan would eventually need both tombstones and versioned entries, not just for change data capture but to improve several other aspects; unfortunately that was in December 2010 and never became a priority, but the benefits are clear. The complexities which have put off such plans lie in the "garbage collection", aka the need to not grow the history without bounds, and have to drop or compact history. So I'm definitely sold on the need to add a certain amount of history, but we need to define how much of this history is expected to be held. In short, what's the ultimate goal? I see two main but different options intertwined: - allow to synchronize the *final state* of a replica - inspect specific changes For the first case, it would be enough for us to be able to provide a "squashed history" (as in Git squash), but we'd need to keep versioned shapshots around and someone needs to tell you which ones can be garbage collected. For example when a key is: written, updated, updated, deleted since the snapshot, we'll send only "deleted" as the intermediary states are irrelevant. For the second case, say the goal is to inspect fluctuations of price variations of some item, then the intermediary states are not irrelevant. Which one will we want to solve? Both? Personally the attempt of solving the second one seems like a huge pivot of the project, the current data-structures and storage are not designed for this. I see the value of such benefits, but maybe Infinispan is not the right tool for such a problem. I'd prefer to focus on the benefits of the squashed history, and have versioned entries soon, but even in that case we need to define which versions need to be kept around, and how garbage collection / vacuuming is handled. This can be designed to be transparent to the client: handled as an internal implementation detail which we use to improve performance of Infinispan itself, or it can be exposed to clients to implement change data capture, but in this case we need to track which clients are still going to need an older snapshot; this has an impact as clients would need to be registered, and has a significant impact on the storage strategies. Within Kafka the log compaction strategies are configurable; I have no experience with Kafka but the docs seem to suggest that it's most often used to provide the last known value of each key. That would be doable for us, but Kafka also does allow optionally for wider scope retention strategies: can we agree that that would not be an option with Infinispan? If not, these goals need to be clarified. My main concern is that if we don't limit the scope of which capabilities we want Infinispan to provide, it risks to become the same thing as Kafka, rather than integrating with it. I don't think we want to pivot all our storage design into efficiently treating large scale logs. In short, I'd like to see an agreement that analyzing e.g. fluctuations in stock prices would be a non-goal, if these are stored as {"stock name", value} key/value pairs. One could still implement such a thing by using a more sophisticated model, just don't expect to be able to see all intermediary values each entry has ever had since the key was first used. # Commenting on specific proposals On ID generation: I'd definitely go with IDs per segment rather than IDs per key for the purpose of change data capture. If you go with independent IDs per key, the client would need to keep track of each version of each entry, which has an high overhead and degree of complexity for the clients. On the other hand, we already guarantee that each segment is managed by a single primary owner, so attaching the "segment transaction id" to each internal entry being changed can be implemented efficiently by Infinispan. Segment ownership handoff needs to be highly consistent during cluster topology changes, but that requirement already exists; we'd just need to make sure that this monotonic counter is included during the handoff of the responsibility as primary owner of a segment. Thanks, Sanne On 12 December 2016 at 13:58, Gustavo Fernandes <gustavo(a)infinispan.org&gt; wrote: > > > On Fri, Dec 9, 2016 at 9:13 AM, Radim Vansa <rvansa(a)redhat.com&gt; wrote: >> >> But introducing globally monotonous counter means that >> there will be a single contention point. > > > I wonder if the trade-off of Flake Ids [1] could be acceptable for this use > case. > > [1] http://yellerapp.com/posts/2015-02-09-flake-ids.html > > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

On 15 Dec 2016, at 11:18, Gustavo Fernandes <gustavo(a)infinispan.org&gt; wrote: On Thu, Dec 15, 2016 at 9:54 AM, Emmanuel Bernard <emmanuel(a)hibernate.org <mailto:emmanuel@hibernate.org>> wrote: The goal is as followed: allow to collect all changes to push them to Debezium and thus Kafka. This need does not require to remember all changes since the beginning of time in Infinispan. Just enough to: - let Kafka catchup assuming it is the bottleneck - let us not lose a change in Kafka when it happened in Infinispan (coordinator, owner, replicas dying) The ability to read back history would then be handled by the Debezium / Kafka tail, not infinispan itself. Having an embedded Debezium connector pushing everything to Kafka sounds cool, but what impact would it bring to the other stream consumers: * Remote listeners, which is supported in several clients apart from Java * Continuous Queries (the same) * Spark Stream * Other eventual 3rd party stream processors: Apache Flick, Storm, etc.

You mentioned that "The ability to read back history would then be handled by the Debezium / Kafka tail, not infinispan itself", my question was how the other consumers would get access to that history.

The goal is as followed: allow to collect all changes to push them to Debezium and thus Kafka. This need does not require to remember all changes since the beginning of time in Infinispan. Just enough to: - let Kafka catchup assuming it is the bottleneck - let us not lose a change in Kafka when it happened in Infinispan (coordinator, owner, replicas dying) The ability to read back history would then be handled by the Debezium / Kafka tail, not infinispan itself. Check my email on this tread from Dec 9th. > On 12 Dec 2016, at 16:13, Sanne Grinovero <sanne(a)infinispan.org&gt; wrote: > > I'm reading many clever suggestions for various aspects of such a > system, but I fail to see a clear definition of the goal. > >> From Randall's opening email I understand how MySQL does this, but > it's an example and I'm not sure which aspects are implementation > details of how MySQL happens to accomplish this, and which aspects are > requirements for the Infinispan enhancement proposals. > > I remember a meeting with Manik Surtani, Jonathan Halliday and Mark > Little, whose outcome was a general agreement that Infinispan would > eventually need both tombstones and versioned entries, not just for > change data capture but to improve several other aspects; > unfortunately that was in December 2010 and never became a priority, > but the benefits are clear. > The complexities which have put off such plans lie in the "garbage > collection", aka the need to not grow the history without bounds, and > have to drop or compact history. > > So I'm definitely sold on the need to add a certain amount of history, > but we need to define how much of this history is expected to be held. > > In short, what's the ultimate goal? I see two main but different > options intertwined: > - allow to synchronize the *final state* of a replica > - inspect specific changes > > For the first case, it would be enough for us to be able to provide a > "squashed history" (as in Git squash), but we'd need to keep versioned > shapshots around and someone needs to tell you which ones can be > garbage collected. > For example when a key is: written, updated, updated, deleted since > the snapshot, we'll send only "deleted" as the intermediary states are > irrelevant. > For the second case, say the goal is to inspect fluctuations of price > variations of some item, then the intermediary states are not > irrelevant. > > Which one will we want to solve? Both? > Personally the attempt of solving the second one seems like a huge > pivot of the project, the current data-structures and storage are not > designed for this. I see the value of such benefits, but maybe > Infinispan is not the right tool for such a problem. > > I'd prefer to focus on the benefits of the squashed history, and have > versioned entries soon, but even in that case we need to define which > versions need to be kept around, and how garbage collection / > vacuuming is handled. > This can be designed to be transparent to the client: handled as an > internal implementation detail which we use to improve performance of > Infinispan itself, or it can be exposed to clients to implement change > data capture, but in this case we need to track which clients are > still going to need an older snapshot; this has an impact as clients > would need to be registered, and has a significant impact on the > storage strategies. > > Within Kafka the log compaction strategies are configurable; I have no > experience with Kafka but the docs seem to suggest that it's most > often used to provide the last known value of each key. That would be > doable for us, but Kafka also does allow optionally for wider scope > retention strategies: can we agree that that would not be an option > with Infinispan? If not, these goals need to be clarified. > > My main concern is that if we don't limit the scope of which > capabilities we want Infinispan to provide, it risks to become the > same thing as Kafka, rather than integrating with it. I don't think we > want to pivot all our storage design into efficiently treating large > scale logs. > > In short, I'd like to see an agreement that analyzing e.g. > fluctuations in stock prices would be a non-goal, if these are stored > as {"stock name", value} key/value pairs. One could still implement > such a thing by using a more sophisticated model, just don't expect to > be able to see all intermediary values each entry has ever had since > the key was first used. > > # Commenting on specific proposals > > On ID generation: I'd definitely go with IDs per segment rather than > IDs per key for the purpose of change data capture. If you go with > independent IDs per key, the client would need to keep track of each > version of each entry, which has an high overhead and degree of > complexity for the clients. > On the other hand, we already guarantee that each segment is managed > by a single primary owner, so attaching the "segment transaction id" > to each internal entry being changed can be implemented efficiently by > Infinispan. > Segment ownership handoff needs to be highly consistent during cluster > topology changes, but that requirement already exists; we'd just need > to make sure that this monotonic counter is included during the > handoff of the responsibility as primary owner of a segment. > > Thanks, > > Sanne > > > > > > On 12 December 2016 at 13:58, Gustavo Fernandes <gustavo(a)infinispan.org&gt; wrote: >> >> >> On Fri, Dec 9, 2016 at 9:13 AM, Radim Vansa <rvansa(a)redhat.com&gt; wrote: >>> >>> But introducing globally monotonous counter means that >>> there will be a single contention point. >> >> >> I wonder if the trade-off of Flake Ids [1] could be acceptable for this use >> case. >> >> [1] http://yellerapp.com/posts/2015-02-09-flake-ids.html >> >> >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

On 15 Dec 2016, at 16:09, Sanne Grinovero <sanne(a)infinispan.org&gt; wrote: Thanks Randall, those clarifications have been great. Emmanuel: some of your statements conflict with Randall's clarifications and with the feasibility points I've been pointing at. You say "collect *all* changes". I've been questioning that Infinispan can not keep *all* changes around for a given single key;

I understand we'd allow clients to retrieve streams of changes persisted into Kafka, but we need to be clear that we won't be handling *all* changes to Kafka (nor to Debezium),

so the magic these can do is somewhat limited. They can certainly expand on the capabilities that Infinispan would provide on its own, but some of the use cases which Gustavo mentioned would not be suitable.

In short, what's the ultimate goal?

On 12 Dec 2016, at 14:58, Gustavo Fernandes <gustavo(a)infinispan.org&gt; wrote: On Fri, Dec 9, 2016 at 9:13 AM, Radim Vansa <rvansa(a)redhat.com&gt; wrote: But introducing globally monotonous counter means that there will be a single contention point. I wonder if the trade-off of Flake Ids [1] could be acceptable for this use case.

[1] http://yellerapp.com/posts/2015-02-09-flake-ids.html _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev