Below is an update on the WildFly NoSQL integration project. The goal is for deployed applications to have access to NoSQL databases (via Hibernate OGM or native APIs). Items 1-4, should be finished in our first pass, with as much of the others items as we can do as well.

1. connection management will deal with obtaining NoSQL connections for application use. - borrow/share Hibernate OGM connection configuration setup code - authentication integration - support transport level security

2. CDI programming simplifications will make it easy to inject NoSQL data into your application classes. - https://github.com/antoinesd/javaee-nosql is initial idea 3. You will easily get a native NoSQL connection from the specified NoSQL profile and use the native NOSQL (Cassandra/MongoDB/other) API in your application.

4. You will also be able to easily use Hibernate OGM with the defined NoSQL profiles (exactly how is TBD but will be awesome :-). - Hibernate OGM static module is included. - need to align with OGM dependencies (e.g. Hibernate ORM + other dependencies). - as mentioned above, OGM already has some connection setup code, which might be good to share for WildFly + standalone NoSQL use. - once WildFly has a common NoSQLSource (not a DataSource) that OGM can use, OGM will be enhanced to use it. 5. How best for the WildFly NoSQL subsystem to be optional? - Is it enough to not run the wildfly/testsuite/nosql tests by default? - Or do we need to start a separate https://github.com/wildfly/nosql project for the NoSQL subsystems? 6. transaction enlistment

7. compensating transactions

8. runtime application monitoring

9. How soon can we make an evaluation distribution available for use on OpenStack/OpenShift? - Would be great if we could do some load testing with all NoSQL components. - Would be great if we could enable others to also test. 10. Are there any problems with our WildFly NoSQL subsystem injecting MongoDatabase connections via: @Resource(lookup = "java:jboss/mongodb/test") MongoDatabase db; - No @Resource support expected for standalone Java, TBD is whether a runtime library can be used. - Any problems expected on other EE application servers if this approach becomes popular?

11. WIP topic branch is at https://github.com/scottmarlow/wildfly/tree/nosql-dev9. Note that every once in a while, commits are squashed and pushed to nosql-devN+1. 12. Add proper unit tests - multi-threaded NoSQL access to show that works at all. - use NoSQL from different EE components (e.g. JAX-RS). - other use cases that represent how NoSQL could be used from WildFly.

Hi, On 05/04/2016 02:16 PM, Scott Marlow wrote: > Below is an update on the WildFly NoSQL integration project. The goal > is for deployed applications to have access to NoSQL databases (via > Hibernate OGM or native APIs). Items 1-4, should be finished in our > first pass, with as much of the others items as we can do as well. > Some general comments: - Good start :) - Maybe refactor from org.jboss.as to org.wildfly

- Maybe use version identifiers in the package name since NoSQL APIs change a lot

- Add some additional asserts to the test cases to verify input / output from the stores > 1. connection management will deal with obtaining NoSQL connections for > application use. > > - borrow/share Hibernate OGM connection configuration setup code > - authentication integration > - support transport level security > Connection management is a huge area to tackle. I think it is best to start with what each NoSQL implementation offer.

The big question is if the connections should be truly managed which would give the option to no-op dangerous API calls out. There aren't any spec requirements for this, so it is really a matter of how the APIs should be presented.

You may have to proxy some of the classes in order to pass the security information into the stores. I think people would expect to use WildFly security domains for this (SubjectFactory/Subject).

> 2. CDI programming simplifications will make it easy to inject NoSQL > data into your application classes. > > - https://github.com/antoinesd/javaee-nosql is initial idea > > 3. You will easily get a native NoSQL connection from the specified > NoSQL profile and use the native NOSQL (Cassandra/MongoDB/other) API in > your application. > I think this is key, as the APIs change almost per release.

> 4. You will also be able to easily use Hibernate OGM with the defined > NoSQL profiles (exactly how is TBD but will be awesome :-). > - Hibernate OGM static module is included. > - need to align with OGM dependencies (e.g. Hibernate ORM + other > dependencies). > - as mentioned above, OGM already has some connection setup code, > which might be good to share for WildFly + standalone NoSQL use. > - once WildFly has a common NoSQLSource (not a DataSource) that OGM > can use, OGM will be enhanced to use it. > > 5. How best for the WildFly NoSQL subsystem to be optional? > - Is it enough to not run the wildfly/testsuite/nosql tests by default? > - Or do we need to start a separate https://github.com/wildfly/nosql > project for the NoSQL subsystems? > > 6. transaction enlistment > This will vary for each store, and there are many loopholes that you may want to plug, f.ex. sharing a connection between 2 transactions. For 1-phase, you will have to insert a LocalXAResource instance into the transaction when the "connection" is obtained, and return it on the boundary (afterCompletion). Same deal for 2-phase basically. The LocalXAResource implementation will of course be different for each store. You may want a transaction option for the stores that supports this such that people can choose the "level" of enlistment (ala jta="false"). One task I see is that people will have access to the transactional methods in the API, and you don't want them to call these methods in their apps unless the transaction setting allows this. I think you can leave out the corner-cases in the 1st iteration, like deferred enlistment (get connection, start transaction). > 7. compensating transactions > Another huge area. I would let the use-cases from WildFly/Swarm drive this. A simple way to start is to have a SPI that apps can implement in case the transaction is MARKED_FOR_ROLLBACK (hooks to an enlisted XAResource).

> 8. runtime application monitoring > As each store is different I would only look into what the subsystem can expose, plus whatever the store can display.

On Thu, May 5, 2016 at 7:48 PM, Jesper Pedersen <jesper.pedersen(a)redhat.com <mailto:jesper.pedersen@redhat.com>> wrote: - Maybe refactor from org.jboss.as <http://org.jboss.as> to org.wildfly - I think you can leave out the "subsystem" part of the package name not really, proper name of would be org.wildfly.extension.name-of-extension for more see https://developer.jboss.org/wiki/WildFlyNamingPolicy

_______________________________________________ wildfly-dev mailing list wildfly-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/wildfly-dev

Hi, Below is an update on the WildFly NoSQL integration project. The goal is for deployed applications to have access to NoSQL databases (via Hibernate OGM or native APIs). Items 1-4, should be finished in our first pass, with as much of the others items as we can do as well. 1. connection management will deal with obtaining NoSQL connections for application use. - borrow/share Hibernate OGM connection configuration setup code - authentication integration - support transport level security 2. CDI programming simplifications will make it easy to inject NoSQL data into your application classes. - https://github.com/antoinesd/javaee-nosql is initial idea 3. You will easily get a native NoSQL connection from the specified NoSQL profile and use the native NOSQL (Cassandra/MongoDB/other) API in your application. 4. You will also be able to easily use Hibernate OGM with the defined NoSQL profiles (exactly how is TBD but will be awesome :-). - Hibernate OGM static module is included. - need to align with OGM dependencies (e.g. Hibernate ORM + other dependencies). - as mentioned above, OGM already has some connection setup code, which might be good to share for WildFly + standalone NoSQL use. - once WildFly has a common NoSQLSource (not a DataSource) that OGM can use, OGM will be enhanced to use it. 5. How best for the WildFly NoSQL subsystem to be optional? - Is it enough to not run the wildfly/testsuite/nosql tests by default? - Or do we need to start a separate https://github.com/wildfly/nosql project for the NoSQL subsystems? 6. transaction enlistment 7. compensating transactions 8. runtime application monitoring 9. How soon can we make an evaluation distribution available for use on OpenStack/OpenShift? - Would be great if we could do some load testing with all NoSQL components. - Would be great if we could enable others to also test. 10. Are there any problems with our WildFly NoSQL subsystem injecting MongoDatabase connections via: @Resource(lookup = "java:jboss/mongodb/test") MongoDatabase db; - No @Resource support expected for standalone Java, TBD is whether a runtime library can be used. - Any problems expected on other EE application servers if this approach becomes popular? 11. WIP topic branch is at https://github.com/scottmarlow/wildfly/tree/nosql-dev9. Note that every once in a while, commits are squashed and pushed to nosql-devN+1. 12. Add proper unit tests - multi-threaded NoSQL access to show that works at all. - use NoSQL from different EE components (e.g. JAX-RS). - other use cases that represent how NoSQL could be used from WildFly. Feedback/help is welcome! Thanks, Scott _______________________________________________ wildfly-dev mailing list wildfly-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/wildfly-dev

That reminds me that we wanted to make the connection to JDG as simple as MongoDB and the like. (i.e. a URL scheme more or less). I'm talking about JDG remote here even though local would be a nice to have. How hard would that be?

Emmanuel On Wed 2016-05-04 14:16, Scott Marlow wrote: > Hi, > > Below is an update on the WildFly NoSQL integration project. The goal > is for deployed applications to have access to NoSQL databases (via > Hibernate OGM or native APIs). Items 1-4, should be finished in our > first pass, with as much of the others items as we can do as well. > > 1. connection management will deal with obtaining NoSQL connections for > application use. > > - borrow/share Hibernate OGM connection configuration setup code > - authentication integration > - support transport level security > > 2. CDI programming simplifications will make it easy to inject NoSQL > data into your application classes. > > - https://github.com/antoinesd/javaee-nosql is initial idea > > 3. You will easily get a native NoSQL connection from the specified > NoSQL profile and use the native NOSQL (Cassandra/MongoDB/other) API in > your application. > > 4. You will also be able to easily use Hibernate OGM with the defined > NoSQL profiles (exactly how is TBD but will be awesome :-). > - Hibernate OGM static module is included. > - need to align with OGM dependencies (e.g. Hibernate ORM + other > dependencies). > - as mentioned above, OGM already has some connection setup code, > which might be good to share for WildFly + standalone NoSQL use. > - once WildFly has a common NoSQLSource (not a DataSource) that OGM > can use, OGM will be enhanced to use it. > > 5. How best for the WildFly NoSQL subsystem to be optional? > - Is it enough to not run the wildfly/testsuite/nosql tests by default? > - Or do we need to start a separate https://github.com/wildfly/nosql > project for the NoSQL subsystems? > > 6. transaction enlistment > > 7. compensating transactions > > 8. runtime application monitoring > > 9. How soon can we make an evaluation distribution available for use on > OpenStack/OpenShift? > - Would be great if we could do some load testing with all NoSQL > components. > - Would be great if we could enable others to also test. > > 10. Are there any problems with our WildFly NoSQL subsystem injecting > MongoDatabase connections via: > > @Resource(lookup = "java:jboss/mongodb/test") MongoDatabase db; > > - No @Resource support expected for standalone Java, TBD is whether a > runtime library can be used. > - Any problems expected on other EE application servers if this > approach becomes popular? > > 11. WIP topic branch is at > https://github.com/scottmarlow/wildfly/tree/nosql-dev9. Note that every > once in a while, commits are squashed and pushed to nosql-devN+1. > > 12. Add proper unit tests > - multi-threaded NoSQL access to show that works at all. > - use NoSQL from different EE components (e.g. JAX-RS). > - other use cases that represent how NoSQL could be used from WildFly. > > Feedback/help is welcome! > > Thanks, > Scott > _______________________________________________ > wildfly-dev mailing list > wildfly-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/wildfly-dev

On 10 May 2016, at 16:45, Scott Marlow <smarlow(a)redhat.com&gt; wrote: > That reminds me that we wanted to make the connection to JDG as simple > as MongoDB and the like. (i.e. a URL scheme more or less). > I'm talking about JDG remote here even though local would be a nice to > have. > > How hard would that be? I think that org.wildfly.clustering.dispatcher.CommandDispatcher is for invoking nodes on the WildFly cluster but I'm not sure of the API for invoking JDG remote and whether that is planned to be part of WildFly already, as part of our clustering effort. Could/should JDG access fall into NoSQL integration?

> > 7. compensating transactions > > > > Another huge area. > > I would let the use-cases from WildFly/Swarm drive this. > > A simple way to start is to have a SPI that apps can implement in case > the transaction is MARKED_FOR_ROLLBACK (hooks to an enlisted > XAResource). > > > We implemented those in IJ (even if in much more standard JCA > environment), and maybe we could share and eventually copy/paste some > code. Check out "MongoDB integration test with compensating transactions" https://github.com/scottmarlow/wildfly/commit/9bd67964a9259416add73cac473..., which is a good start. We can work forward from what we have and ideas from IJ.

> Hibernate OGM should still be usable without WF; So maybe there should > be a separate project/repo which defines an SPI to obtain/manage > connections and implementations for different NoSQL stores? Excellent suggestion, perhaps the SPI could be under https://github.com/jboss, which is a common area for sharing. Possible locations for creating the per NoSQL store implementations could be https://github.com/jboss or https://github.com/hibernate or https://github.com/wildfly.

On 11 mai 2016, at 16:02, Scott Marlow <smarlow(a)redhat.com&gt; wrote: >> Hibernate OGM should still be usable without WF; So maybe there should >> be a separate project/repo which defines an SPI to obtain/manage >> connections and implementations for different NoSQL stores? > > Excellent suggestion, perhaps the SPI could be under > https://github.com/jboss, which is a common area for sharing. Possible > locations for creating the per NoSQL store implementations could be > https://github.com/jboss or https://github.com/hibernate or > https://github.com/wildfly. I'm starting to think that this might be way overkill. If we are creating a sub project just to share between 20 and 50 lines of code per provider and the overhead code to abstract property configuration to plus OGM and WF ones, we are losing more than gaining. Thoughts ?

If OGM should be able to work with connections managed by the container, some sort of "neutral" SPI is needed IMO. Otherwise we'd have to create some sort of abstraction *in* OGM to ensure it still can be used in other environments than WF.

I think best would be to start with a PoC and see things are going to look like. Then we still can decide where it'd be best located or whether it's better to avoid re-use and accept some duplication.

2016-05-12 13:24 GMT+02:00 Emmanuel Bernard <emmanuel(a)hibernate.org <mailto:emmanuel@hibernate.org>>: On 11 mai 2016, at 16:02, Scott Marlow <smarlow(a)redhat.com <mailto:smarlow@redhat.com>> wrote: >> Hibernate OGM should still be usable without WF; So maybe there should >> be a separate project/repo which defines an SPI to obtain/manage >> connections and implementations for different NoSQL stores? > > Excellent suggestion, perhaps the SPI could be under > https://github.com/jboss, which is a common area for sharing. Possible > locations for creating the per NoSQL store implementations could be > https://github.com/jboss or https://github.com/hibernate or > https://github.com/wildfly. I'm starting to think that this might be way overkill. If we are creating a sub project just to share between 20 and 50 lines of code per provider and the overhead code to abstract property configuration to plus OGM and WF ones, we are losing more than gaining. Thoughts ? _______________________________________________ wildfly-dev mailing list wildfly-dev(a)lists.jboss.org <mailto:wildfly-dev@lists.jboss.org> https://lists.jboss.org/mailman/listinfo/wildfly-dev

On 12 May 2016 at 12:24, Emmanuel Bernard <emmanuel(a)hibernate.org&gt; wrote: > > > On 11 mai 2016, at 16:02, Scott Marlow <smarlow(a)redhat.com&gt; wrote: > >>> Hibernate OGM should still be usable without WF; So maybe there should >>> be a separate project/repo which defines an SPI to obtain/manage >>> connections and implementations for different NoSQL stores? >> >> Excellent suggestion, perhaps the SPI could be under >> https://github.com/jboss, which is a common area for sharing. Possible >> locations for creating the per NoSQL store implementations could be >> https://github.com/jboss or https://github.com/hibernate or >> https://github.com/wildfly. > > I'm starting to think that this might be way overkill. If we are creating a sub project just to share between 20 and 50 lines of code per provider and the overhead code to abstract property configuration to plus OGM and WF ones, we are losing more than gaining. > > Thoughts ? I agree it's overkill, and have an alternative proposal: Hibernate OGM should define an interface which is appropriate for its own consumption; the Wildfly NoSQL subssystem can have its own interface so to not depend on OGM, but they would be somewhat similar for each given NoSQL technology we intend to support in this way.

Then JipiJapa can inject an adaptor into the OGM boostrap phase, delegating from one to the other. So only the OGM specific JipiJapa module would need to depend on both interfaces. If this dependency is not desirable either, then I think we can live with a non-typesafe generic provider of things.

Thanks, Sanne

On 12 May 2016 at 12:24, Emmanuel Bernard <emmanuel(a)hibernate.org&gt; wrote: > > > On 11 mai 2016, at 16:02, Scott Marlow <smarlow(a)redhat.com&gt; wrote: > >>> Hibernate OGM should still be usable without WF; So maybe there should >>> be a separate project/repo which defines an SPI to obtain/manage >>> connections and implementations for different NoSQL stores? >> >> Excellent suggestion, perhaps the SPI could be under >> https://github.com/jboss, which is a common area for sharing. Possible >> locations for creating the per NoSQL store implementations could be >> https://github.com/jboss or https://github.com/hibernate or >> https://github.com/wildfly. > > I'm starting to think that this might be way overkill. If we are creating a sub project just to share between 20 and 50 lines of code per provider and the overhead code to abstract property configuration to plus OGM and WF ones, we are losing more than gaining. > > Thoughts ? I agree it's overkill, and have an alternative proposal: Hibernate OGM should define an interface which is appropriate for its own consumption; the Wildfly NoSQL subssystem can have its own interface so to not depend on OGM, but they would be somewhat similar for each given NoSQL technology we intend to support in this way. Then JipiJapa can inject an adaptor into the OGM boostrap phase, delegating from one to the other. So only the OGM specific JipiJapa module would need to depend on both interfaces. If this dependency is not desirable either, then I think we can live with a non-typesafe generic provider of things.

I think that we will move forward now and prototype the transaction enlistment for ACID NoSQL databases. This is important to nail down now, as we would like forward application compatibility. We need to choose now, whether MySLSB.myMethod() performs the Neo4j database update immediately when (#4) tx.close() is called or when the JTA transaction completes. By now, I mean that we need to decide before a release of WildFly that contains the NoSQL integration is released, so that we know that future releases will be compatible. I don't think that transaction enlistment should be allowed for BASE NoSQL databases, which is why I mentioned ACID above. With BASE databases, the application code is responsible for dealing with dirty data. As such, I think it would be misleading to enlist BASE operations into a JTA transaction. This means that BASE NoSQL database profiles will not allow 1-phase/2-phase transaction attribute to be specified. We started to write some design notes [1] for NoSQL integration. We will update the design notes after hearing more feedback on the above. Scott [1] https://developer.jboss.org/wiki/DesignNotesForNoSQLIntegrationOnWildFly

Hi, On 07/08/2016 12:24 PM, Scott Marlow wrote: > The NoSQL prototype now has transaction enlistment for Neo4j (one phase > wrapper). > > Test example link [1] shows a CMT test bean method You should add a comment that tx.success() / tx.failure() is tied to the out-come of the JTA transaction. And tx.close() is done in afterCompletion.

> and [2] shows a BMT test bean method. > This test case highlights the problem of checking for enlistment in 2 places, as the Session is obtained before the transaction is started, and there is no way to get the session reference from the transaction instance. So, you can manage the transaction lifecycle, but not the session lifecycle.

> Currently, in a JTA transaction the Neo4j Session survives ending the > JTA transaction. It might be better to close the Neo4j Session when the > JTA transaction ends, which would require the application to get a new > Session, this would ensure that applications don't forget to close the > session, with the down side that with BMT, application code would need > to call the Neo4j Driver.session(), to get a new Session instance after > each JTA transaction ends. After a session is closed, there is no > guarantee that another session can be obtained, as the next > Driver.session() may time out if the internal pool of Driver sessions, > is empty and the configured time limit > (neo4j.driver.acquireSessionTimeout) is exceeded (defaults to 30 > seconds). So, there are costs to automatically closing Neo4j Sessions > at JTA transaction end time. Although, you could say that closing the > Session, after each transaction, gives other threads a change to use the > Session that isn't really closed, but instead is really returned to the > internal Driver sessions pool. As you are enlisting the Neo4J resources into the transaction, I think that you should define driver.session() as the enlistment barrier, and therefore also close the session in afterCompletion.

That makes it more clear in what scenarios the resources are managed. In JDBC you inject the DataSource instance, and obtain a Connection, so it would be parallel to this case Driver vs. Session. This means that people needs to be more careful in the BMT case to get the Session within an active transaction. JCA has CachedConnectionManager to verify these cases, so you could add a similar concept.

On 07/12/2016 11:36 AM, Scott Marlow wrote: >> As you are enlisting the Neo4J resources into the transaction, I think >> that you should define driver.session() as the enlistment barrier, and >> therefore also close the session in afterCompletion. > > I pushed an update [1] that includes auto closing the session, when the > transaction ends. > > I didn't add a Synchronization yet, to handle closing the session when > the transaction manager doesn't call XAResourceWrapper.commit/rollback. > I agree that will introduce an extra layer of safety to ensure that > resources are closed. > I think you can use the XAResource instance for that at the moment.

>> >> That makes it more clear in what scenarios the resources are managed. >> >> In JDBC you inject the DataSource instance, and obtain a Connection, so >> it would be parallel to this case Driver vs. Session. >> >> This means that people needs to be more careful in the BMT case to get >> the Session within an active transaction. JCA has >> CachedConnectionManager to verify these cases, so you could add a >> similar concept. > > Attempts to call Neo4j transaction/session "close", within the JTA > transaction are ignored. How about calls to "close" after the JTA > transaction ends, should they also be ignored? Hopefully the underlying implementation will ignore multiple close() calls, and throw an exception on any other call.

You will have to test the behaviour of each NoSQL store. > Is there a JCA > equivalent of ignoring "close" after the transaction ends? I think that > Hibernate ORM is or was, closing DataSource's after the transaction > ends, so I assume that JCA ignores "close" after the JTA transaction > ends. > You probably mean Connection.close() -- if all connections are closed in afterCompletion then the connection is returned to the pool. Otherwise, it waits until ref count becomes 0, e.g. c.close() after the transaction has ended. http://www.ironjacamar.org/doc/roadto12/txtracking.html Best regards, Jesper

On 13 juil. 2016, at 19:16, Gunnar Morling <gunnar(a)hibernate.org&gt; wrote: Hi Scott, I had a short look at the tests you pointed to; in the one using BMT, why is it that the UserTransaction *and* the Neo4j TX API are used? I'd have expected it to be only one *or* the other? In the one using CMT, there are also calls to the Neo4j TX API which apparently are ignored. What's the rationale behind that? Wouldn't it make more sense to raise an exception, similar to when EntityManager#getTransaction() is invoked for a JTA entity manager? In the CMT case, did you consider to inject the Neo4j Session instead of Driver into the bean, again akin to injection of JTA entity managers? All in all, I'm wondering whether things couldn't be handled analogous to the usage of entity managers: * For a JTA-enabled NoSQL persistence unit, obtain the Session via injection (its lifecycle is controlled by the container and bound to the managed transaction), any manual interaction with the Neo4j TX API is forbidden, instead the container manages transactions or the user is (via UserTransaction) * For a non-JTA-enabled NoSQL persistence unit, obtain the Driver via injection, and the user is in charge of Session lifecycle and transaction control I hope that'd make sense for actually transactional stores such as Neo4j, for non-TX stores such as MongoDB thinks look different. --Gunnar 2016-07-13 16:25 GMT+01:00 Scott Marlow <smarlow(a)redhat.com&gt;: > > > On 07/13/2016 10:44 AM, Jesper Pedersen wrote: > > On 07/12/2016 11:36 AM, Scott Marlow wrote: > >>> As you are enlisting the Neo4J resources into the transaction, I think > >>> that you should define driver.session() as the enlistment barrier, and > >>> therefore also close the session in afterCompletion. > >> > >> I pushed an update [1] that includes auto closing the session, when the > >> transaction ends. > >> > >> I didn't add a Synchronization yet, to handle closing the session when > >> the transaction manager doesn't call XAResourceWrapper.commit/rollback. > >> I agree that will introduce an extra layer of safety to ensure that > >> resources are closed. > >> > > > > I think you can use the XAResource instance for that at the moment. > > Yes, we are doing this currently. > > > > >>> > >>> That makes it more clear in what scenarios the resources are managed. > >>> > >>> In JDBC you inject the DataSource instance, and obtain a Connection, so > >>> it would be parallel to this case Driver vs. Session. > >>> > >>> This means that people needs to be more careful in the BMT case to get > >>> the Session within an active transaction. JCA has > >>> CachedConnectionManager to verify these cases, so you could add a > >>> similar concept. > >> > >> Attempts to call Neo4j transaction/session "close", within the JTA > >> transaction are ignored. How about calls to "close" after the JTA > >> transaction ends, should they also be ignored? > > > > Hopefully the underlying implementation will ignore multiple close() > > calls, and throw an exception on any other call. > > isOpen is currently returning false instead of throwing an exception. > We are ignoring multiple close() calls and do throw an exception on any > other call. > > > > > You will have to test the behaviour of each NoSQL store. > > > >> Is there a JCA > >> equivalent of ignoring "close" after the transaction ends? I think that > >> Hibernate ORM is or was, closing DataSource's after the transaction > >> ends, so I assume that JCA ignores "close" after the JTA transaction > >> ends. > >> > > > > You probably mean Connection.close() -- if all connections are closed in > > afterCompletion then the connection is returned to the pool. Otherwise, > > it waits until ref count becomes 0, e.g. c.close() after the transaction > > has ended. > > > > http://www.ironjacamar.org/doc/roadto12/txtracking.html > > > > Best regards, > > Jesper > > > _______________________________________________ > wildfly-dev mailing list > wildfly-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/wildfly-dev _______________________________________________ wildfly-dev mailing list wildfly-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/wildfly-dev

Hi Gunnar/Emmanuel, The Ne04j profile can either be "transaction=1pc" (JTA transaction enlisted) or "transaction=none" (will not be enlisted into JTA transaction). I think this covers what Emmanuel suggested (I didn't get Emmanuel's email yet for some unknown reason). It sounds like we aren't yet connecting on what the WF Neo4j profile is doing, so I'll respond below and we can continue from there. :) On 07/13/2016 01:16 PM, Gunnar Morling wrote: > Hi Scott, > > I had a short look at the tests you pointed to; in the one using BMT, > why is it that the UserTransaction *and* the Neo4j TX API are used? I'd > have expected it to be only one *or* the other? With both container managed transactions and bean managed transactions (BMT), a JTA transaction is in control of the transaction. With BMT, UserTransaction is used by the application code to start/end the transaction. In either case (BMT or CMT), the Neo4j TX API is controlled by the JTA transaction. If there isn't an active JTA transaction, the Ne04j TX API is in control of the Neo4j transaction. With one caveat, that the Ne04j transaction should be ended by the application code, before entering into a JTA transaction (since it won't be joined into the JTA transaction). Users can also disable the JTA transaction enlistment by using a NoSQL profile with "transaction=none", instead of "transaction=1pc". > > In the one using CMT, there are also calls to the Neo4j TX API which > apparently are ignored. With CMT + BMT, the JTA transaction is in control, when the JTA transaction ends, the Neo4j transaction ends as well and not before then. If the application calls the Neo4j TX API, our TXProxy ignores calls to TX.success/failure/close, other calls are passed through to the underlying Neo4j Transaction object (e.g. run(String statement)), since that is where the work actually occurs. When the JTA transaction commits, the underlying Neo4j tx.success() + tx.close() are called. If the JTA transaction rolls back, the underlying Neo4j tx.failure() + tx.close() are called. > What's the rationale behind that? To enlist the Neo4j transaction into the JTA transaction via a one phase wrapper. > Wouldn't it > make more sense to raise an exception, similar to when > EntityManager#getTransaction() is invoked for a JTA entity manager? If the NoSQL profile is JTA enabled, it should be enlisted into the JTA transaction, allowing the application to control all of the database operations that occur within the JTA transaction. We also have the equivalent of JPA RESOURCE_LOCAL, via NoSQL profiles that are not JTA enabled ("transaction=none" in the profile definition). > > In the CMT case, did you consider to inject the Neo4j Session instead of > Driver into the bean, again akin to injection of JTA entity managers? It is possible that we could also look at TransactionScoped injection of sessions, but not sure yet. We started with (RequestScoped) Session injection but commented that out [1] when we added JTA enlistment. TransactionScoped injection of Sessions could be yet another way to write CMT/BMT code. > > All in all, I'm wondering whether things couldn't be handled analogous > to the usage of entity managers: > > * For a JTA-enabled NoSQL persistence unit, obtain the Session via > injection (its lifecycle is controlled by the container and bound to the > managed transaction), any manual interaction with the Neo4j TX API is > forbidden, instead the container manages transactions or the user is > (via UserTransaction) > * For a non-JTA-enabled NoSQL persistence unit, obtain the Driver via > injection, and the user is in charge of Session lifecycle and > transaction control Well, you haven't yet seen the above responses yet, I think the TransactionScoped Session injection idea is good but also introduces a different way to write CMT/BMT code for Ne04j. I disagree on making the current (WF NoSQL/Ne04j TX enlistment) approach illegal, as I don't yet see a problem with the current approach. Are there difficulties with Hibernate OGM that you anticipate with either approach of JTA TX enlistment for Neo4j? > > I hope that'd make sense for actually transactional stores such as > Neo4j, for non-TX stores such as MongoDB thinks look different. Agreed, for MongoDB, we treat that as "transaction=none", meaning no JTA transaction enlistment for MongoDB. > > --Gunnar > Scott [1] https://github.com/scottmarlow/wildfly/blob/nosql-dev9/nosql/neo4j/src/ma...

The choice of ignoring those calls, is made against the alternative of throwing an exception. The reason for ignoring the neo4j transactional calls, is to be more compatible with existing ne04j code that may be using the neo4j transactional code, with the goal that the JTA transaction controls the underlying ne04j transaction. Neo4j transactional code also runs the statements against the neo4j transaction class, instead of the neo4j session class.

> > The choice of ignoring those calls, is made against the alternative of > throwing an exception. The reason for ignoring the neo4j transactional > calls, is to be more compatible with existing ne04j code that may be > using the neo4j transactional code, with the goal that the JTA > transaction controls the underlying ne04j transaction. Neo4j > transactional code also runs the statements against the neo4j > transaction class, instead of the neo4j session class. > In a JTA transaction, instead of ignoring calls to org.neo4j.driver.v1.Transaction.failure(), we could call setRollbackOnly() on the JTA transaction. So, the existing neo4j code can influence a JTA rollback. IMO, this would improve our compatibility with existing neo4j.

On 07/19/2016 04:17 AM, Emmanuel Bernard wrote: > On Mon 2016-07-18 12:20, Scott Marlow wrote: >> Thanks for the feedback. >> >> On 07/18/2016 09:24 AM, Emmanuel Bernard wrote: >>> I'm lost. >>> >>> What is the expected code a developer will write ? >>> >>> If that' something along the lines of: >>> >>> jta.beginTx() // declaratively or imperatively >>> neo4j.beginTx() >>> ... >>> neo4j.commit() >>> jta.commit() //declaratively or imperatively >> >> They are not limited to writing the above code. In addition, the developer >> can also avoid calling neo4j.beginTx()/neo4j.commit(). Something like: >> >> jta.beginTx() // declaratively or imperatively >> session = driver.session() // internally does session.beginTransaction() >> ... >> jta.commit() //declaratively or imperatively >> >> The calls to neo4j.beginTx()/commit() are ignored. >> >> The choice of ignoring those calls, is made against the alternative of >> throwing an exception. The reason for ignoring the neo4j transactional >> calls, is to be more compatible with existing ne04j code that may be using >> the neo4j transactional code, with the goal that the JTA transaction >> controls the underlying ne04j transaction. Neo4j transactional code also >> runs the statements against the neo4j transaction class, instead of the >> neo4j session class. > > You mean having existing code work with the new JTA transaction net? > I don't think you can safely support that. What if the code does > > @Transactional > public void doThings() { > session = drive.session(); > session.beginTransaction(); // should be: Transaction transaction = session.beginTransaction(); > try { > // do things > catch (Exception e) { > // swallow exception because we will fix things > session.rollback(); // should be: transaction.failure(); > session.beginTransaction(); transaction = session.beginTransaction(); > // do something else > session.commit(); > } > } > > Such an existing code won't work if JTA in involved and > session.beginTrnasaction() / commit() are ignored. With the current prototype, the above existing code won't work as originally intended (with regard to what the original developer had in mind) with JTA enlistment. > > So only code written with the idea of the JTA net in mind will be safe. Clearly, any existing neo4j code that expects to fail the current neo4j transaction, then start a new neo4j transaction to fix things, will not work in the JTA net. As the fixing of things in the JTA net, needs to occur before marking the JTA transaction for rollback only *or* by fixing things after the JTA transaction rolls back. Are there any other holes that come to mind, with regard to existing neo4j transaction code that won't work as original intended in the JTA net? > >>> >>> then I find it very user unfriendly to force the code to call both Tx >>> APIs. It should be one or the other. >> >> Do you really want to rewrite your neo4j code to avoid the neo4j tx APIs >> when enlisted into a JTA transaction? Or should the neo4j tx API calls be >> ignored when enlisted into a JTA transaction (as we are doing now)? > > See my above example of already existing code. > > For new code that you want to run both in SE and Wildfly, that's an > interesting question but how would you do it in practice? A try catch > but rethrowing the exception in case JTA is involved and needs to be > notified of the tx rollback? The developer could rewrite their "workaround tx failure" code to avoid further calls to the neo4j tx api, but this is kind of bad as they only should do this when the neo4j transaction is enlisted into the JTA transaction, but not when enlistment is disabled. Some neo4j connection profiles will be configured to allow JTA enlistment (transaction=1pc) and some will be configured to not allow JTA enlistment (transaction=none), so we can expect common application code will likely be reused for both JTA + non-JTA. Seems to me that SE is always the same as the WildFly neo4j connection profile with "transaction=none", so we have the same problem with SE + WildFly, trying to run the same user code that is expected to work with JTA enlistment (neo4j connection profile has "transaction=1pc"). I don't know of any CDI magic that helps, as if we add support for inject of a javax.transaction.TransactionScoped neo4j session, that works the same as a session obtained from Driver.session(). > I'm interested to see concrete code because if doable, it will have lots > of constraints and be limited to very specific patterns of code I think. > But again, let's show each other code.

That should help us explore the compatibility with existing code idea and pick which approach to use (e.g. whether explicit neo4j transaction use in a JTA enlisted transaction should throw an exception or ignore the explicit transaction use as we are doing now in the prototype). Should we also post on https://groups.google.com/forum/#!forum/neo4j to ask for input on JTA enlistment? > > Emmanuel > _______________________________________________ wildfly-dev mailing list wildfly-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/wildfly-dev

I think that this wouldn’t be a common case, but it is still possible that user could inject drivers from two different profiles and invoke them in the same transaction. In this case two Neo4jXAResourceImpl instances would be enlisted to the transaction and that is not a recommended use case because atomicity cannot be guaranteed. There is a bit more info about it in section 3.2.1.1. of http://narayana.io/docs/product/index.html <http://narayana.io/docs/product/index.html>;.

On 20 Jul 2016, at 14:59, Scott Marlow <smarlow(a)redhat.com&gt; wrote: On 07/20/2016 06:02 AM, Gytis Trikleris wrote: > Scott, > > I think that this wouldn’t be a common case, but it is still possible > that user could inject drivers from two different profiles and invoke > them in the same transaction. In this case two Neo4jXAResourceImpl > instances would be enlisted to the transaction and that is not a > recommended use case because atomicity cannot be guaranteed. There is a > bit more info about it in section 3.2.1.1. > of http://narayana.io/docs/product/index.html. I'm trying to understand how the description in 3.2.1.1 applies to Neo4jXAResourceImpl (implements org.jboss.tm.LastResource and not com.arjuna.ats.jta.resources.LastResourceCommitOptimisation). By default, will WildFly allow more than one org.jboss.tm.LastResource to be enlisted into the JTA transaction at a time? When you say above that its possible for two Neo4jXAResourceImpl's to be injected, would that only be if com.arjuna.ats.jta.allowMultipleLastResources is changed to true? Thanks for pointing this case out! Scott > > Gytis > >> On 19 Jul 2016, at 19:36, Scott Marlow <smarlow(a)redhat.com >> <mailto:smarlow@redhat.com>> wrote: >> >> >> >> On 07/19/2016 10:49 AM, Scott Marlow wrote: >>> >>> >>> On 07/19/2016 04:17 AM, Emmanuel Bernard wrote: >>>> On Mon 2016-07-18 12:20, Scott Marlow wrote: >>>>> Thanks for the feedback. >>>>> >>>>> On 07/18/2016 09:24 AM, Emmanuel Bernard wrote: >>>>>> I'm lost. >>>>>> >>>>>> What is the expected code a developer will write ? >>>>>> >>>>>> If that' something along the lines of: >>>>>> >>>>>> jta.beginTx() // declaratively or imperatively >>>>>> neo4j.beginTx() >>>>>> ... >>>>>> neo4j.commit() >>>>>> jta.commit() //declaratively or imperatively >>>>> >>>>> They are not limited to writing the above code. In addition, the >>>>> developer >>>>> can also avoid calling neo4j.beginTx()/neo4j.commit(). Something like: >>>>> >>>>> jta.beginTx() // declaratively or imperatively >>>>> session = driver.session() // internally does >>>>> session.beginTransaction() >>>>> ... >>>>> jta.commit() //declaratively or imperatively >>>>> >>>>> The calls to neo4j.beginTx()/commit() are ignored. >>>>> >>>>> The choice of ignoring those calls, is made against the alternative of >>>>> throwing an exception. The reason for ignoring the neo4j transactional >>>>> calls, is to be more compatible with existing ne04j code that may >>>>> be using >>>>> the neo4j transactional code, with the goal that the JTA transaction >>>>> controls the underlying ne04j transaction. Neo4j transactional >>>>> code also >>>>> runs the statements against the neo4j transaction class, instead of the >>>>> neo4j session class. >>>> >>>> You mean having existing code work with the new JTA transaction net? >>>> I don't think you can safely support that. What if the code does >>>> >>>> @Transactional >>>> public void doThings() { >>>> session = drive.session(); >>>> session.beginTransaction(); >>> // should be: >>> Transaction transaction = session.beginTransaction(); >>>> try { >>>> // do things >>>> catch (Exception e) { >>>> // swallow exception because we will fix things >>>> session.rollback(); >>> // should be: >>> transaction.failure(); >>>> session.beginTransaction(); >>> transaction = session.beginTransaction(); >>>> // do something else >>>> session.commit(); >>>> } >>>> } >>>> >>>> Such an existing code won't work if JTA in involved and >>>> session.beginTrnasaction() / commit() are ignored. >>> >>> With the current prototype, the above existing code won't work as >>> originally intended (with regard to what the original developer had in >>> mind) with JTA enlistment. >>> >>>> >>>> So only code written with the idea of the JTA net in mind will be safe. >>> >>> Clearly, any existing neo4j code that expects to fail the current neo4j >>> transaction, then start a new neo4j transaction to fix things, will not >>> work in the JTA net. As the fixing of things in the JTA net, needs to >>> occur before marking the JTA transaction for rollback only *or* by >>> fixing things after the JTA transaction rolls back. >>> >>> Are there any other holes that come to mind, with regard to existing >>> neo4j transaction code that won't work as original intended in the >>> JTA net? >>> >>>> >>>>>> >>>>>> then I find it very user unfriendly to force the code to call both Tx >>>>>> APIs. It should be one or the other. >>>>> >>>>> Do you really want to rewrite your neo4j code to avoid the neo4j tx >>>>> APIs >>>>> when enlisted into a JTA transaction? Or should the neo4j tx API >>>>> calls be >>>>> ignored when enlisted into a JTA transaction (as we are doing now)? >>>> >>>> See my above example of already existing code. >>>> >>>> For new code that you want to run both in SE and Wildfly, that's an >>>> interesting question but how would you do it in practice? A try catch >>>> but rethrowing the exception in case JTA is involved and needs to be >>>> notified of the tx rollback? >>> >>> The developer could rewrite their "workaround tx failure" code to avoid >>> further calls to the neo4j tx api, but this is kind of bad as they only >>> should do this when the neo4j transaction is enlisted into the JTA >>> transaction, but not when enlistment is disabled. Some neo4j connection >>> profiles will be configured to allow JTA enlistment (transaction=1pc) >>> and some will be configured to not allow JTA enlistment >>> (transaction=none), so we can expect common application code will likely >>> be reused for both JTA + non-JTA. Seems to me that SE is always the >>> same as the WildFly neo4j connection profile with "transaction=none", so >>> we have the same problem with SE + WildFly, trying to run the same user >>> code that is expected to work with JTA enlistment (neo4j connection >>> profile has "transaction=1pc"). >>> >>> I don't know of any CDI magic that helps, as if we add support for >>> inject of a javax.transaction.TransactionScoped neo4j session, that >>> works the same as a session obtained from Driver.session(). >>> >>>> I'm interested to see concrete code because if doable, it will have lots >>>> of constraints and be limited to very specific patterns of code I think. >>>> But again, let's show each other code. >> >> I found some examples in the Neo4j Java driver unit tests. I started a >> gist [1] that lists a few of the test cases that I thought were >> interesting for this discussion about "compatibility of existing Neo4j >> tx handling code in the JTA net". >> >> In actual applications, the use cases will not be so simple. Instead, >> we will see many different patterns mixed over several lines of >> application library code, however, each unit case noted is presenting a >> different example that would probably not run as is, when enlisted in a >> JTA transaction. >> >> This doesn't mean it is bad to enlist Neo4j transactions into a JTA >> transaction, it just means that it is bad assume that a Neo4j tx works >> the same when enlisted in a JTA transaction. >> >> I think that a challenge here for us, is to deduce what our rules should >> be when enlisting Neo4j transactions into a JTA transaction, with regard >> to what the application can expect and shouldn't expect. >> >> Scott >> >> [1] https://gist.github.com/scottmarlow/2779d21653e8466d85e38c1d62d4bb1e >> >>> >>> That should help us explore the compatibility with existing code idea >>> and pick which approach to use (e.g. whether explicit neo4j transaction >>> use in a JTA enlisted transaction should throw an exception or ignore >>> the explicit transaction use as we are doing now in the prototype). >>> >>> Should we also post on https://groups.google.com/forum/#!forum/neo4j >>> <https://groups.google.com/forum/#%21forum/neo4j> to >>> ask for input on JTA enlistment? >>> >>>> >>>> Emmanuel >>>> >>> _______________________________________________ >>> wildfly-dev mailing list >>> wildfly-dev(a)lists.jboss.org <mailto:wildfly-dev@lists.jboss.org> >>> https://lists.jboss.org/mailman/listinfo/wildfly-dev >>> >> _______________________________________________ >> wildfly-dev mailing list >> wildfly-dev(a)lists.jboss.org <mailto:wildfly-dev@lists.jboss.org> >> https://lists.jboss.org/mailman/listinfo/wildfly-dev >

> The transactional properties will be different in each NoSQL store, > f.ex. in Neo4J tx.success() / tx.failure() doesn't do anything before > tx.close() is called. So having transaction="1pc" (and "2pc") is a way > to unify transactional integration across different NoSQL stores. We are taking a closer look at different Neo4j examples, to better understand the pros/cons of allowing applications to call tx.success/tx.failure in the JTA transaction. I think that the advantage of enlisting Neo4j into a JTA transaction is that it makes it easier for the application developer to coordinate updating multiple resources within the same transaction.