[jbossts-issues] [JBoss JIRA] (JBTM-2920) Investigate the use of transactions within a reactive microservices environment

[ https://issues.jboss.org/browse/JBTM-2920?page=com.atlassian.jira.plugin.... ] Tom Jenkinson updated JBTM-2920: -------------------------------- Description: Investigate the use of a transaction manager when used within a reactive microservices environment[1][2]. h2. Motivation Reactive microservice architecture is a trending architectural pattern. The reactive architecture could simply be defined as a system where messages are sent to a jms queue or a specialized event storage and other services are registered to, for being notified when an event is generated to be able to react on it. Synchronous calls from a service to a service are not meant to be a good practice in such system. Here all the communication between services is done with message passing through the event storage. Such a system is expected to decouple data [3] where each microservice manages data, which needs for working, on its own. Still there are cases where some data coordination is needed. As the all communication is asynchronous it may be that we need to modify the transaction manager away from synchronous operations. One possibly good match seems to be the Saga transactions[4] and eventual consistency[5][6] but handling of them is fitting to the message driven system. The way of understanding Saga in microservice world could be overviewed at [15]. Narayana transaction manager is good in managing Saga transactions where having long history experience from running it for WS[7] or managing consistency in NoSQL databases[8]. But Narayana is designed to manage transactions in synchronous way. The transaction context is normally passed along a RPC call where the transaction is finished after such synchronous call is returned back to the caller. h2. Steps to go The point here is to investigate on the topic of usage Saga in reactive architecture, gather working solutions - especially in Java world (some references eg. [9][10][11][12][13][14]), get experience in how they are working and on top of this experience considering implementation of the reactive approach with Narayana. It may be that Narayana requires some modification to execute optimally for such task and then it’s up to reconsideration if some greenfield implementation should be started. As the last step there should be created some sample project (ie. quickstart) where the usage of the implementation would be shown. With that we expect usage of projects in Red Hat portfolio. We can expect the need for an event storage system (a.k.a. A scalable transaction log) - as an original thought let’s consider the AMQ7 message broker or the Infinispan. The sample project would be probably implemented with usage of the Vert.x. h2. Expected output The expected output of this effort is * A research document with state of the art. ** Overview of what are the problem of the synchronous/blocking approach. ** What are the other approaches that people have to this problem, * A proof-of-concept implementation ** Preferably with use of Narayana transaction manager prepare a service capable to manage transactions in world of reactive microservices * An example/quickstart showing this in practical terms ** A proof that a transaction manager can be made to work in an asynchronous environment. For example if you have a Vert.x event and it has to commit a transaction. The handler for that message currently does sync ops with transaction manager and that blocks. One of the goals for the project is to prevent blocking of threads in an async environment. h2. Resources [1] [https://www.youtube.com/watch?v=STKCRSUsyP0|https://www.youtube.com/watch... 2017, The Many Meanings of Event-Driven Architecture, Martin Fowler) [2] [http://www.oreilly.com/programming/free/reactive-microservices-architectu...] (Reactive Microservices Architecture) [3] [http://blog.christianposta.com/microservices/the-hardest-part-about-micro...] (The Hardest Part About Microservices: Your Data) [4] [https://www.cs.cornell.edu/andru/cs711/2002fa/reading/sagas.pdf] (Sagas paper) [5] [http://queue.acm.org/detail.cfm?id=2462076] (Eventual Consistency Today: Limitations, Extensions, and Beyond) [6] [http://queue.acm.org/detail.cfm?id=3025012] (Life Beyond Distributed Transactions) [7] [http://narayana.io//docs/project/index.html#d0e14874] (Narayana doc, WS-Business Activity) [8] [http://jbossts.blogspot.cz/2014/05/bringing-transactional-guarantees-to.html] (Bringing Transactional Guarantees to MongoDB) [9] [https://www.youtube.com/watch?v=0UTOLRTwOX0] (Distributed Sagas: A Protocol for Coordinating Microservices - Caitie McCaffrey - JOTB17) [10] [https://www.youtube.com/watch?v=YPbGW3Fnmbc] (Using sagas to maintain data consistency in a microservice architecture by Chris Richardson) [11] [http://www.axonframework.org] [12] [http://eventuate.io] [13] [https://docs.particular.net/nservicebus/sagas] (.NET NServiceBus Saga) [14] [https://www.youtube.com/watch?v=Rm8n-H6zI1k] (Dr. Roland Kuhn: Reactive Design Patterns, Akka) [15] [http://microservices.io/patterns/data/saga.html] (Pattern: Saga) was: Investigate the use of a transaction manager when used within a reactive microservices environment[1][2]. h2. Motivation Reactive microservice architecture is a trending architectural pattern. The reactive architecture could simply be defined as a system where messages are sent to a jms queue or a specialized event storage and other services are registered to, for being notified when an event is generated to be able to react on it. Synchronous calls from a service to a service are not meant to be a good practice in such system. Here all the communication between services is done with message passing through the event storage. Such a system is expected to decouple data [3] where each microservice manages data, which needs for working, on its own. Still there are cases where some data coordination is needed. As the all communication is asynchronous it may be that we need to modify the transaction manager away from synchronous operations. The good fit seems to be the Saga transactions[4] and eventual consistency[5][6] but handling of them is fitting to the message driven system. The way of understanding Saga in microservice world could be overviewed at [15]. Narayana transaction manager is good in managing Saga transactions where having long history experience from running it for WS[7] or managing consistency in NoSQL databases[8]. But Narayana is designed to manage transactions in synchronous way. The transaction context is normally passed along a RPC call where the transaction is finished after such synchronous call is returned back to the caller. h2. Steps to go The point here is to investigate on the topic of usage Saga in reactive architecture, gather working solutions - especially in Java world (some references eg. [9][10][11][12][13][14]), get experience in how they are working and on top of this experience considering implementation of the reactive approach with Narayana. It may be that Narayana requires some modification to execute optimally for such task and then it’s up to reconsideration if some greenfield implementation should be started. As the last step there should be created some sample project (ie. quickstart) where the usage of the implementation would be shown. With that we expect usage of projects in Red Hat portfolio. We can expect the need for an event storage system (a.k.a. A scalable transaction log) - as an original thought let’s consider the AMQ7 message broker or the Infinispan. The sample project would be probably implemented with usage of the Vert.x. h2. Expected output The expected output of this effort is * A research document with state of the art. ** Overview of what are the problem of the synchronous/blocking approach. ** What are the other approaches that people have to this problem, * A proof-of-concept implementation ** Preferably with use of Narayana transaction manager prepare a service capable to manage transactions in world of reactive microservices * An example/quickstart showing this in practical terms ** A proof that a transaction manager can be made to work in an asynchronous environment. For example if you have a Vert.x event and it has to commit a transaction. The handler for that message currently does sync ops with transaction manager and that blocks. One of the goals for the project is to prevent blocking of threads in an async environment. h2. Resources [1] [https://www.youtube.com/watch?v=STKCRSUsyP0|https://www.youtube.com/watch... 2017, The Many Meanings of Event-Driven Architecture, Martin Fowler) [2] [http://www.oreilly.com/programming/free/reactive-microservices-architectu...] (Reactive Microservices Architecture) [3] [http://blog.christianposta.com/microservices/the-hardest-part-about-micro...] (The Hardest Part About Microservices: Your Data) [4] [https://www.cs.cornell.edu/andru/cs711/2002fa/reading/sagas.pdf] (Sagas paper) [5] [http://queue.acm.org/detail.cfm?id=2462076] (Eventual Consistency Today: Limitations, Extensions, and Beyond) [6] [http://queue.acm.org/detail.cfm?id=3025012] (Life Beyond Distributed Transactions) [7] [http://narayana.io//docs/project/index.html#d0e14874] (Narayana doc, WS-Business Activity) [8] [http://jbossts.blogspot.cz/2014/05/bringing-transactional-guarantees-to.html] (Bringing Transactional Guarantees to MongoDB) [9] [https://www.youtube.com/watch?v=0UTOLRTwOX0] (Distributed Sagas: A Protocol for Coordinating Microservices - Caitie McCaffrey - JOTB17) [10] [https://www.youtube.com/watch?v=YPbGW3Fnmbc] (Using sagas to maintain data consistency in a microservice architecture by Chris Richardson) [11] [http://www.axonframework.org] [12] [http://eventuate.io] [13] [https://docs.particular.net/nservicebus/sagas] (.NET NServiceBus Saga) [14] [https://www.youtube.com/watch?v=Rm8n-H6zI1k] (Dr. Roland Kuhn: Reactive Design Patterns, Akka) [15] [http://microservices.io/patterns/data/saga.html] (Pattern: Saga) -- This message was sent by Atlassian JIRA (v7.2.3#72005)