Mark Little [http://community.jboss.org/people/marklittle] created the discussion "Re: Remoting Transport Transaction Inflow Design Discussion" To view the discussion, visit: http://community.jboss.org/message/621427#621427 -------------------------------------------------------------- Some initial thoughts/questions: a) what do you mean by "unaddressed issues - transaction timeout control"? b) since this is distributed transactions (context propagation, commit protocol termination etc.) have you considered the scope of the testing that needs to be done in order to ensure this works in *all* cases? c) "any prepared but incomplete transactions have to be recovered manually"? d) in case 2, where the client is within the same container as a tm instance, which tm is it trying to control/invoke? The remote one (in the other container instance) or the local one? Because we seem to have gone from controlling transactions (begin, commit, rollback) in the first scenario, to really doing distributed transactions in the second scenario. As you can imagine, these are very different situations. e) what about interoposition, particularly in the case where A calls B calls C calls D, and maybe even D calls A? -------------------------------------------------------------- Reply to this message by going to Community [http://community.jboss.org/message/621427#621427] Start a new discussion in JBoss Transactions Development at Community [http://community.jboss.org/choose-container!input.jspa?contentType=1&...]

Jonathan Halliday [http://community.jboss.org/people/jhalliday] created the discussion "Re: Remoting Transport Transaction Inflow Design Discussion" To view the discussion, visit: http://community.jboss.org/message/621445#621445 -------------------------------------------------------------- re: ( 1 )  Whilst the remote UserTransaction model is easy to implement, it's tricky to document. It behaves in an intuitive fashion only for a very limited, albeit common, set of use cases. For more complex scenarios its inherent limitations manifest in ways that can be confusing to users. The cost of this solution must be evaluated not just in terms of implementation time, but in documentation and ongoing support cost. re: ( 2 ) JCA inflow was either designed for propagation to leaf nodes only, or incredibly badly thought out. Either way, the result is a model that simply isn't capable enough for a full distributed transaction solution. So, once again you're going to wind up with unintuitive behaviour and limitations for some use cases. For example, you can't register more than one resource with an inflowed transaction, you can't allow a context to be flowed into a node by more than one route and you need to register each subordinate node in the parent for recovery purposes. So, whilst this may provide a faster alternative to RMI/IIOP for simple use cases, it's not going to be an acceptable substitute for more advanced cases. Offering it alongside the traditional RMI/IIOP model is going to have implications beyond just the initial engineering, so the discussion may require input from docs, support and QE too. I'm pretty sure that e.g. support will tell you it is unacceptable to ship a solution that may require manual transaction cleanup. We've had a small number of corner cases in JTA that suffered that limitation and eliminating them and the support load they generate has been a high priority for the transaction development work. Intentionally introducing new ones is definitely in the category of Bad Ideas. -------------------------------------------------------------- Reply to this message by going to Community [http://community.jboss.org/message/621445#621445] Start a new discussion in JBoss Transactions Development at Community [http://community.jboss.org/choose-container!input.jspa?contentType=1&...]

Mark Little [http://community.jboss.org/people/marklittle] created the discussion "Re: Remoting Transport Transaction Inflow Design Discussion" To view the discussion, visit: http://community.jboss.org/message/621462#621462 -------------------------------------------------------------- So does this answer my question c) too? Are you saying that "any prepared but incomplete transactions have to be recovered manually" is only a reference to heuristic outcomes? -------------------------------------------------------------- Reply to this message by going to Community [http://community.jboss.org/message/621462#621462] Start a new discussion in JBoss Transactions Development at Community [http://community.jboss.org/choose-container!input.jspa?contentType=1&...]

David Lloyd [http://community.jboss.org/people/dmlloyd] created the discussion "Re: Remoting Transport Transaction Inflow Design Discussion" To view the discussion, visit: http://community.jboss.org/message/621463#621463 -------------------------------------------------------------- I mean that I haven't addressed the issue of transaction timeout control. Keeping in mind that this is nowhere near the only process of this complexity to be tested - and no, don't trot out "it's more complex than you think" unless you want to enumerate specific cases (which will probably then be appropriated into additional tests) - I think we'd follow the same approach we'd follow for testing other things.  We'd unit test the protocol of course, and test to ensure that the implementation matches the specification, and verify that the protocol handlers on either "end" forward to the proper APIs. If you're asking "can we write automated tests to prove the validity of the approach", well no we can't; we're just coding against existing contracts, and if those are "wrong", well, that fix needs to happen in a different context. That's just another way of saying we don't have any special, magical auto-recovery "stuff" that isn't provided by the transaction coordinator (which might well have some magical auto-recovery "stuff").  There might be a better way to express that. In case 1, the client has no TM and it uses a remote UserTransaction interface to directly control the remote TM.  In case 2, the client is using the local TM to control transactions, and is treating the remote TM as an enrolled resource into the current transaction. Case 1 cannot be made to work when a local TM is present without adding some notion in the EE layer to determine whether it should use the local UserTransaction or the remote one.  This is possible but is a possibly significant amount of work. Theoretically each successive "step" will treat the TM of the subsequent "step" as a participating resource.  As to D calling A, that will only work if the TM is clever enough to figure out what's happening (I don't see why it wouldn't as the Xid should, well, identify the transaction so A should recognize its own; but that's why we're having this discussion). -------------------------------------------------------------- Reply to this message by going to Community [http://community.jboss.org/message/621463#621463] Start a new discussion in JBoss Transactions Development at Community [http://community.jboss.org/choose-container!input.jspa?contentType=1&...]

Mark Little [http://community.jboss.org/people/marklittle] created the discussion "Re: Remoting Transport Transaction Inflow Design Discussion" To view the discussion, visit: http://community.jboss.org/message/621469#621469 -------------------------------------------------------------- I don't know about the resource perspective at this point, but I know it would be a lot less than writing another distributed transactions protocol ;-) I ported the C++ and Java transaction services to pretty much every C++ or Java ORB on the planet by 2005. As to performance? That really depends what we're trying to achieve and when. It's an option. Whether it's the right option in the long term and for all use cases, I don't think anyone can say precisely at this point. -------------------------------------------------------------- Reply to this message by going to Community [http://community.jboss.org/message/621469#621469] Start a new discussion in JBoss Transactions Development at Community [http://community.jboss.org/choose-container!input.jspa?contentType=1&...]

Mark Little [http://community.jboss.org/people/marklittle] created the discussion "Re: Remoting Transport Transaction Inflow Design Discussion" To view the discussion, visit: http://community.jboss.org/message/621474#621474 -------------------------------------------------------------- "If a transaction is prepared (XATerminator.prepare()) and then the connection is lost, then the transaction is "stuck" in the prepared state - we aren't going to do anything about that at the protocol level.  But presumably the transaction coordinator would be able to automatically recover once the resource was made available again, as the transaction isn't "busted", it's just not fully completed." Well that really depends upon what the participant does or, even worse, what other participants do in the same transaction. For instance, it could decide to autonomously roll back, and in which case we're in heuristic land. So the transaction could well be "busted" in as much as we now risk not having an atomic outcome. Some error codes from XA allow for the transaction manager to periodically retry operations on the RM. A heuristic isn't one of them. "If the same scenario exists but the commit was received and acted upon (but it failed), then I guess this is what you refer to as a heuristic outcome, and, well we're not going to deal with that either; presumably our recovery tooling will work exactly the same way in this case as it does in the case where the same thing happens to your database connection." No, in the scenario you just mentioned the recovery subsystem would retry and either get back a "didn't you get my message" response (c.f. XTS) or eventually decide that the RM did commit and it missed the message. Remember that we're using presumed abort semantics here. -------------------------------------------------------------- Reply to this message by going to Community [http://community.jboss.org/message/621474#621474] Start a new discussion in JBoss Transactions Development at Community [http://community.jboss.org/choose-container!input.jspa?contentType=1&...]

David Lloyd [http://community.jboss.org/people/dmlloyd] created the discussion "Re: Remoting Transport Transaction Inflow Design Discussion" To view the discussion, visit: http://community.jboss.org/message/621483#621483 -------------------------------------------------------------- Okay, well we've proposed more than once to move off of JacORB.  I'd be in favor for sure.  But I think that's a peripheral topic at this point. -------------------------------------------------------------- Reply to this message by going to Community [http://community.jboss.org/message/621483#621483] Start a new discussion in JBoss Transactions Development at Community [http://community.jboss.org/choose-container!input.jspa?contentType=1&...]

David Lloyd [http://community.jboss.org/people/dmlloyd] created the discussion "Re: Remoting Transport Transaction Inflow Design Discussion" To view the discussion, visit: http://community.jboss.org/message/621489#621489 -------------------------------------------------------------- Exactly.  It's just a question of how these get propagated, which seems somewhat outside of the core of this solution.  It's only mentioned because it's in the SPIs. Well, it's always good to have a set of existing projects to draw test scenarios from.  But otherwise I don't think this is directly relevant to the discussion: unless you're saying "we must test these 200 different scenarios before I let you type 'git commit'".  We need high quality, detailed tests for every subsystem.  For example having thorougly tested transactions doesn't do us a lot of good if, for example, our JPA implementation or HornetQ or something was writing corrupt data.  I mean everything needs thorough testing.  Just the fact that these other projects have lots of tests covering recovery doesn't mean that those tests are necessary, and on the other hand, there may be many scenarios unaccounted-for in these tests as well.  AS is riddled with highly complex systems that need detailed testing. If we use an SPI with a documented contract, it is not unreasonable to expect that contract to be met by its implementation.  If the contract is +not+ met by the implementation, yeah that's a bug, but saying that it's the responsibility of every project consuming that SPI to verify that its implementation(s) meet the SPI contract is crazy.  Yeah we may introduce a test here or there to catch regression in the target project, but even this is not strictly necessary as the target project should be doing this! In this particular case (solution 2 that is), we're specifying an implementation for XAResource, a transport for it, and an endpoint which controls XATerminator; this says to me that our tests can be limited in scope to testing this mechanism from end to end.  As I said if we have other projects we can draw recovery scenarios from, that's fine, and we will do so.  I don't know what else to tell you. Yes, that is my assumption, as we are using existing propagation mechanisms. The problem is that we'd need some way to control which kind of UserTransaction is pulled from JNDI and thus injected into EE components.  This can depend on what the user intends to do with it; thus we'd need to isolate many use cases and figure out what level this should be done at (deployment? component? server-wide?), and we need to do some analysis to determine where and how the remote server connection(s) should be specified and associate the two somehow.  We're basically choosing between TMs on a per-operation basis.  This type of configuration is unprecedented as far as I know - I think the analysis would take as long as the implementation, if not longer.  Because it is not known exactly how this should look, I can't say how much effort this is going to be other than "lots". Yeah but keep in mind that we're dealing in a strict hierarchy here, there are no peers.  The transaction isn't so much "distributed" as it is "controlled"; caller always dominates callee.  This if D calls A the behavior I'd +expect+ would be that A would treat the imported work as a different or subordinate transaction; it need not really have any direct knowledge that the two are related since the D→A relationship is controlled by D, and the C→D relationship is controlled by C, etc. If the D→A outcome is in doubt then it's up to D to resolve that branch, not A.  But that's just my ignoramus opinion. When it comes to reality, this situation is extremely unlikely to occur even in the weirdest situations I've ever heard of.  The reason is that if you've got two nodes invoking on each other, it is highly likely that they are within the same "tier", which greatly increases the likelihood that they could simply run JTS and be done. Here's what I consider to be a likely, real-world scenario: Host A runs a thin client which uses the "solution 1" mechanism to control the transaction when it talks to Host B. Host B runs a "front" tier which is isolated by firewall.  This tier has one or more local transactional databases or caches, and a local TM.  The services running on B also perform EJB invocations on Host C. Host C is the "rear" tier separated from B by one or more layer of firewall, and maybe even a public network.  B talks to C via remoting, using "solution 2" to propagate transactions to it, using the client/server style of invocation. Host C participates in a peer-to-peer relationship with other services on Hosts D, E, and F in the same tier, using Remoting or IIOP but using JTS to coordinate the transaction at this level since C, D, E, and F all mutulally execute operations on one another (and possibly each consume local resources) in a distributed object graph style of invocation. Note you can substitue A and B with an EIS and everything should be exactly the same (except that recovery processes would be performed by the EIS rather than by B's TM). Everything I understand about transaction processing (which is definitely at least as much as a "joe user") says that there's no reason this shouldn't "just work".  And we should be able to utilize existing transaction recovery mechanisms as well. -------------------------------------------------------------- Reply to this message by going to Community [http://community.jboss.org/message/621489#621489] Start a new discussion in JBoss Transactions Development at Community [http://community.jboss.org/choose-container!input.jspa?contentType=1&...]

Jonathan Halliday [http://community.jboss.org/people/jhalliday] created the discussion "Re: Remoting Transport Transaction Inflow Design Discussion" To view the discussion, visit: http://community.jboss.org/message/621512#621512 -------------------------------------------------------------- Sure, explaining the chain of reasoning behind that one is easy: Red Hat ships products and offers support on them. That support is fixed price based on SLA, not proportional to the number of tickets filed. On the other hand, support costs scale as a function of the number of issues reported. Thus the less support work we have to do, the lower our cost and the higher our profit. Intentionally building and shipping something we know is going to increase support load is contra-survival and therefore a Bad Idea. yup, I can give you a some of them too: 1) The 'client' is actually another AS instance, either of the same or earlier vintage, doing JNDI lookup of UserTransaction against a remote AS7. 2) The client wants to talk to two remote AS instances in the same tx. 3) The client is an environment that has its own UserTransaction implementation. This is actually just a more general version of case 1). but in which you can't use tricks like patching the client side lookup to return your actual UserTransaction instead of the remote proxy. 4) you want to support load balancing or failover for the client-server connection. yup, although it's really pretty obvious: The JCA inflow API uses an Xid as a poor man's transaction propagation context. Xids were designed only for control flow between a transaction manager and a resource manager, not for use in multi-level trees. The JCA has no provision for allowing subordinates to create new branches in the global transaction. For that it would have to pass in a mask of free bits in the bqual array as well as the Xid to the subordinate. Indeed the JCA expressly prohibits the container handling the inflow from altering the Xid. It has to remain immutable because without any knowledge of which bits can safely be mutated, the container can't guarantee to generate uniq Xids, a property which is required by the spec. The XA spec expects that each resource manager (give or take its XAResource's isSameRM implementation) gets its own branch i.e. uniq Xid. With inflowed Xids you can't generate new Xids to meet that expectation, you have to use the inflowed one verbatim. That causes problems with the state machine for the XA protocol lifecycle, as it's tied to the Xid. For example, if the inflowed tx is used to connect to two resource managers, you can't recover from crashes cleanly as the recovery mechanism is tracking state on the assumption that the Xid belongs to at most one RM and once it has cleaned that one up it's done.  Actually on further thought even an upper limit of one is optimistic - the Xid contains the node Id of the originating parent and that parent may connect to the same resource manager, in which case it's going to incorrectly manage the lifecycle because it can't distinguish the XAResource representing the subordinate tx from the one representing the RM as they have the same Xid. That last case is an artifact of our implementation rather than the spec though. yup, I can give you one for that too - the afterCompletions run relative to the commit in the local node where they are registered, which may actually be before the commit in another node and not correctly reflect heuristics outcomes or be suitable for triggering subsequent steps in a process that depend on running after commits in the other nodes. Likewise beforeCompletions run relative to the prepare in the local node, thus may run after a prepare in another node. In the best case that's merely inefficient, in the worst case, where resource managers are shared, it causes a flush of cached data to occur after a prepare, which will fail.  It that's not complicated enough for you, take the inflowed transaction context and make a transactional call back to the originating parent server. fun, fun. It's a deficiency in the JCA spec, see above. The spec assumes the inflowed container is a leaf node i.e. RM, not a subordinate coordinator. There are some hacky things we can potentially do to work around that limitation in the spec without outright breaking compliance. They were on my list of things to do in the transactions upstream, but I seem to be a bit busy with AS integration issues instead :-) Not at all. MDBs don't normally run in inflowed transactions. The server hosting the MDB container starts a top level transaction, enlists the JMS as a resource manager and additionally enlists any resource managers the MDB calls e.g. a database. It's a flat structure, not a hierarchic one. sure, but they are the small number of outcomes that result from one or more of the players not behaving in accordance with the spec e.g. resource managers making autonomous outcome decisions. We don't automatically do anything about those because we simply can't - that's the point at which the spec basically says 'give up, throw a heuristic and let a human deal with the mess'.  You're talking about the much more numerous expected failure cases that can be handled under automatically under the spec. Indeed exactly the kinds of run of the mill system failures a distributed transaction protocol is designed to protect a user against.  Intentionally shipping a non spec compliant XAResource implementation that will result in a support case for many of those common failures is borderline business suicide, see above. Really? Got a test case for that? Other than the one a certain competitor wrote and we soundly refuted as FUD? Because I've got an extensive test suite that shows no such outcomes. Well, except for MS SQLServer and mysql, neither of which is fully XA compliant at present. Ensuring clean transaction completion in crash situations is exactly what the transaction manager is for after all. -------------------------------------------------------------- Reply to this message by going to Community [http://community.jboss.org/message/621512#621512] Start a new discussion in JBoss Transactions Development at Community [http://community.jboss.org/choose-container!input.jspa?contentType=1&...]

Mark Little [http://community.jboss.org/people/marklittle] created the discussion "Re: Remoting Transport Transaction Inflow Design Discussion" To view the discussion, visit: http://community.jboss.org/message/621521#621521 -------------------------------------------------------------- Understood. Humour me and put them here explicitly. So what about using one of the existing transports over which transactions are supported, rather than implement yet another one? Of course the performance of, say, HTTP isn't as good as JBR or any binary protocol, but I've yet to see any performance requirements being made against this particular requirement. In fact as far as I can tell, we're talking about the requirement to support a case that we couldn't support in 5/6 based only on the fact that would couldn't support it, not on the fact that we need to support it and with N tx/sec. Yes SOAP/HTTP is equally slow (can be slower) than HTTP, but SOAP/JMS is an option. I suppose if there was enough reason we could even consider a SOAP/JBR, though at that point I'd be first in the queue to recommend removing SOAP from the equation entirely! -------------------------------------------------------------- Reply to this message by going to Community [http://community.jboss.org/message/621521#621521] Start a new discussion in JBoss Transactions Development at Community [http://community.jboss.org/choose-container!input.jspa?contentType=1&...]

David Lloyd [http://community.jboss.org/people/dmlloyd] created the discussion "Re: Remoting Transport Transaction Inflow Design Discussion" To view the discussion, visit: http://community.jboss.org/message/621530#621530 -------------------------------------------------------------- See below. Well that's what I'm talking about: this is the direct functional replacement for the Remoting 2.x-based UnifiedInvoker stuff, and the JRMPInvoker before it.  This is the functionality we're carrying over.  I can tell you with absolute certainty that porting this stuff over directly is not a viable option.  We can discuss the details offline if you like. This is really bleeding over into another topic at this point.  Remoting 3.x is the transport layer that we're using for AS management; the plan was to bring over a JSR-160 implementation plus EJB remote invocation which allows all these things to run over the same channel, and to leverage JBMAR, to get an optimally-performant invocation layer which supports all kinds of security strategies, and works with even the stupidest of firewalls.  In other words, replace the old broken stuff with a combination of a bunch of existing good stuff.  This in turn gives us a nice springboard for nicely supporting EE app-client, getting this multi-tier transaction stuff for free (or at least, that was the plan) and at the same time giving us a shiny new bullet point to throw up against the competitors who have a similar sort of transport already.  Also it gives us a nice path forward to allow EAP 4 and 5 applications (and even applications running on third-party appservers) to talk to EAP 6 applications using this protocol simply by way of a client JAR which is something that has been requested of us more than once.  We +know+ that JBMAR outperforms everything else out there which claims any level of compliance to the serialization spec; we can choose to use it, or not. -------------------------------------------------------------- Reply to this message by going to Community [http://community.jboss.org/message/621530#621530] Start a new discussion in JBoss Transactions Development at Community [http://community.jboss.org/choose-container!input.jspa?contentType=1&...]

Jason Greene [http://community.jboss.org/people/jason.greene] created the discussion "Re: Remoting Transport Transaction Inflow Design Discussion" To view the discussion, visit: http://community.jboss.org/message/621569#621569 -------------------------------------------------------------- While we need to figure out the short term, I really think we should start formulating a plan for our long term direction as part of this discussion. As part of that I think we should explore the possibility of making JTS capable of running on a native transport. -------------------------------------------------------------- Reply to this message by going to Community [http://community.jboss.org/message/621569#621569] Start a new discussion in JBoss Transactions Development at Community [http://community.jboss.org/choose-container!input.jspa?contentType=1&...]

Mark Little [http://community.jboss.org/people/marklittle] created the discussion "Re: Remoting Transport Transaction Inflow Design Discussion" To view the discussion, visit: http://community.jboss.org/message/621638#621638 -------------------------------------------------------------- Definitely. Although it's not quite JTS over some other transport, but I know what you mean. -------------------------------------------------------------- Reply to this message by going to Community [http://community.jboss.org/message/621638#621638] Start a new discussion in JBoss Transactions Development at Community [http://community.jboss.org/choose-container!input.jspa?contentType=1&...]

David Lloyd [http://community.jboss.org/people/dmlloyd] created the discussion "Re: Remoting Transport Transaction Inflow Design Discussion" To view the discussion, visit: http://community.jboss.org/message/626589#626589 -------------------------------------------------------------- Okay it looks to me like we have a plan here, but do we have a path forward?  What do you need from the AS team, if anything? -------------------------------------------------------------- Reply to this message by going to Community [http://community.jboss.org/message/626589#626589] Start a new discussion in JBoss Transactions Development at Community [http://community.jboss.org/choose-container!input.jspa?contentType=1&...]

Jonathan Halliday [http://community.jboss.org/people/jhalliday] created the discussion "Re: Remoting Transport Transaction Inflow Design Discussion" To view the discussion, visit: http://community.jboss.org/message/627264#627264 -------------------------------------------------------------- Sorry, running a bit behind what with having spent the last two work days in a meeting room with lousy connectivity :-( The discussion above is on the basis of 'what would we ideally like to have in the future' per the related conf call. We're not even at the stage of 'we will deliver feature enhancement X by date Y' yet. What we eventually commit to implement will depend on what other requirements we have competing for attention in the same timeframe and we don't have all that information yet. For the AS7.1 release we've already delivered the final TS feature release, so if you want tx over remoting in that you will indeed need to build it on whatever is already there, which basically means JCA tx inflow. -------------------------------------------------------------- Reply to this message by going to Community [http://community.jboss.org/message/627264#627264] Start a new discussion in JBoss Transactions Development at Community [http://community.jboss.org/choose-container!input.jspa?contentType=1&...]

Mark Little [http://community.jboss.org/people/marklittle] created the discussion "Re: Remoting Transport Transaction Inflow Design Discussion" To view the discussion, visit: http://community.jboss.org/message/627453#627453 -------------------------------------------------------------- OK, but are we all in agreement that this is a stop-gap measure and not where we'd like to be in some later release? Also can you (Jason) stick the design discussions around what you will do in this forum, or at least cross post? -------------------------------------------------------------- Reply to this message by going to Community [http://community.jboss.org/message/627453#627453] Start a new discussion in JBoss Transactions Development at Community [http://community.jboss.org/choose-container!input.jspa?contentType=1&...]

Jason Greene [http://community.jboss.org/people/jason.greene] created the discussion "Re: Remoting Transport Transaction Inflow Design Discussion" To view the discussion, visit: http://community.jboss.org/message/627472#627472 -------------------------------------------------------------- That should read 6.1 -------------------------------------------------------------- Reply to this message by going to Community [http://community.jboss.org/message/627472#627472] Start a new discussion in JBoss Transactions Development at Community [http://community.jboss.org/choose-container!input.jspa?contentType=1&...]

Mark Little [http://community.jboss.org/people/marklittle] created the discussion "Re: Remoting Transport Transaction Inflow Design Discussion" To view the discussion, visit: http://community.jboss.org/message/627495#627495 -------------------------------------------------------------- Did you not see Jonathan's response from a few weeks back? There was no response to that either. I'll copy it here: " ClientUserTransaction requires no additional implementation work. The same hooks that supported the earlier implementations of this can be used for the new one too. Transaction context inflow support falls into two parts: whole transaction (gtrid) interposition and branch only (bqual) interposition. For whole transaction interposition, a new subordinate transaction context is created on each node receiving an inflow. Synchronizations are handled purely locally. The JCA inflow API can be used, albeit with semantics which IMO are not spec compliant.  This model would be relatively simple to implement on the transaction manager side, as the existing recovery architecture will mostly still apply. IMO it's of limited utility for users though, as resource managers will see independent transactions and not do any transaction branch coupling. That impacts both functionality and performance. For branch only transaction interposition, subordinate nodes maintain the inflowed gtrid but create new branches within an allocated portion of the bqual state space. This requires information about the allocation of bqual space to be communicated, either by explicit parameter passing for the general case or by encoding in the Xid for the jboss->jboss inflow case. It affords the opportunity for branch coupling in resource managers used by more than one node in the same transaction, but leads to more complicated recovery needs. Specifically recovery can no longer be driven off consideration of the gtrid ownership alone, but must also consider bqual ownership. This naturally requires that the bqual value actually contain node ownership information, which will require a new encoding. On the other hand we probably need one anyhow to communicate delegation of the bqual state space on links where we're working with 3rd party implementations and thus constrained to the JCA api rather than one that could carry additional parameters.  For links where we do control both ends, we need additional methods to support afterCompletion as a separate phase. BeforeCompletion is already available as a separate step on subordinate transactions, although it may need to be even finer grained to allow for JTA 1.1 TSR sync interposition semantics to be transaction global rather than node local. In both models the communication is entirely top down - the coordinator does not exist as a network endpoint as in JTS, as persistent ids are not supported by the remoting transport. This constrains recovery to use the XA recovery scan model rather than the JTS replayCompletion one. One consequence of this is that parent nodes will require to maintain a list of all possible subordinates and have a recovery module plugin for them, but that's probably not an undue burden for most deployment scenarios.  Another consequence is that it's probably going to be better to build it as distributed hooks into JBossJTA rather than a pluggable transport layer for JBossJTS. That should also offer better performance as resource records will get inlined to the tx rather than be separate ostore entries." -------------------------------------------------------------- Reply to this message by going to Community [http://community.jboss.org/message/627495#627495] Start a new discussion in JBoss Transactions Development at Community [http://community.jboss.org/choose-container!input.jspa?contentType=1&...]

Tom Jenkinson [http://community.jboss.org/people/tomjenkinson] created the discussion "Re: Remoting Transport Transaction Inflow Design Discussion" To view the discussion, visit: http://community.jboss.org/message/627601#627601 -------------------------------------------------------------- I agree wholeheartedly with Jonathan and also think we definitely need to take heed on his point re functionality available at each stage being documented accurately and the overhead this will add to EAP. Jason, are we certain this is a feature that needs to be in EAP6 vs EAP6.1? I guess more generally can you clarify your preferred and maximum timescales you would like us to deliver this functionality within. Another point that was raised earlier (or rather a detail of the one surrounding QA) is the question of testing this with remoting itself. Clearly some work will be done on the remoting side to chat with the API we expose so is it realistic to expect this level of testing to be performed in the timescale needed? It sounds like a nice piece of work, but our worry is we may simply not be able to add this feature with enough capability/testing to address your desired use cases in the timescale we anticipate you need this in. That said we are ready to work on this right now so hopefully we can make an effective solution in the time available. -------------------------------------------------------------- Reply to this message by going to Community [http://community.jboss.org/message/627601#627601] Start a new discussion in JBoss Transactions Development at Community [http://community.jboss.org/choose-container!input.jspa?contentType=1&...]

Mark Little [http://community.jboss.org/people/marklittle] created the discussion "Re: Remoting Transport Transaction Inflow Design Discussion" To view the discussion, visit: http://community.jboss.org/message/627615#627615 -------------------------------------------------------------- Tom said: "It sounds like a nice piece of work, but our worry is we may simply not be able to add this feature with enough capability/testing to address your desired use cases in the timescale we anticipate you need this in. That said we are ready to work on this right now so hopefully we can make an effective solution in the time available." Let's all remember that we have a "baking in the community first" mantra here, and for critical (read complex) pieces of code like this, I definitely want to see that baking taken into consideration. So let's be sure to factor that in too! -------------------------------------------------------------- Reply to this message by going to Community [http://community.jboss.org/message/627615#627615] Start a new discussion in JBoss Transactions Development at Community [http://community.jboss.org/choose-container!input.jspa?contentType=1&...]

Mark Little [http://community.jboss.org/people/marklittle] created the discussion "Re: Remoting Transport Transaction Inflow Design Discussion" To view the discussion, visit: http://community.jboss.org/message/627619#627619 -------------------------------------------------------------- I definitely want to see the design happen in the forums so it's all inclusive. As to the steps, I'm pretty sure that Jonathan and Tom already have a plan of attack since this is something we've discussed for a while and most recently last week at the JBossTS f2f. I'll encourage one or other of them to post it here as soon as possible. -------------------------------------------------------------- Reply to this message by going to Community [http://community.jboss.org/message/627619#627619] Start a new discussion in JBoss Transactions Development at Community [http://community.jboss.org/choose-container!input.jspa?contentType=1&...]

Mark Little [http://community.jboss.org/people/marklittle] created the discussion "Re: Remoting Transport Transaction Inflow Design Discussion" To view the discussion, visit: http://community.jboss.org/message/627622#627622 -------------------------------------------------------------- I've mentioned this before, but the overriding issue here is that we will deliver EAP 6 on schedule. If the use cases we need to have in EAP 6 from TS cannot be done by the ideal solution then we probably have no option but to have a stop-gap implementation. We've discussed that on this forum before and elsewhere so I think everyone understands what that would be. On the other hand, if we can skip EAP 6 and go straight to EAP 6.1 and cover the use cases with the ideal solution as outlined, then that gives us the better implementation, more time to bake in the community, more time to QA etc. Yes, testing this is critical. We've seen over the past 20 years that testing the recovery scenarios always take more time than anything else and for good reasons. So we shouldn't underestimate how long this testing will take. -------------------------------------------------------------- Reply to this message by going to Community [http://community.jboss.org/message/627622#627622] Start a new discussion in JBoss Transactions Development at Community [http://community.jboss.org/choose-container!input.jspa?contentType=1&...]

Jason Greene [http://community.jboss.org/people/jason.greene] created the discussion "Re: Remoting Transport Transaction Inflow Design Discussion" To view the discussion, visit: http://community.jboss.org/message/627627#627627 -------------------------------------------------------------- I've been traveling most of the day and I am cutting an AS release and working on a presentation for a JBUG I need to do in a couple hours, so I don't have much time to comment a more detailed reply to the topic (sorry!). Breifly, I just wanted to mention that my primary concern was having a solution in ANY EAP6 release. To me this was an escalated AS + TS integration problem that everyone agreed needed to be fixed, so whatever stop gaps we go with need to be short lived, and 2 years is just too long to have this issue. I realize that getting this into EAP 6.0 was a big stretch, but the work should be able to start now, and as I mentioned in the past fixing EAP6 integration issues is more important than any EAP7 feature. I definitely want to avoid doing a one off patched fork, but after hearing that no changes are allowed to happen in the only release branch that is compatible with EAP6, that appeared to be the only option. We do have resource constraints for 6.0, but 6.1 we have breathing room so can "make things happen". I promise to explain in more detail later, but hopeful that clears up my position somewhat. Thanks. -------------------------------------------------------------- Reply to this message by going to Community [http://community.jboss.org/message/627627#627627] Start a new discussion in JBoss Transactions Development at Community [http://community.jboss.org/choose-container!input.jspa?contentType=1&...]

Tom Jenkinson [http://community.jboss.org/people/tomjenkinson] created the discussion "Re: Remoting Transport Transaction Inflow Design Discussion" To view the discussion, visit: http://community.jboss.org/message/627698#627698 -------------------------------------------------------------- Hi Jason, Thanks for the clarifying the timescales involved. Certainly 6.1 is a comfortable position to aim this functionality at and I for one have breathed a sigh of relief :) As soon as we agree a clear picture of what the functionality looks like we should raise a Jira for this in the JBTM project so we can track the functionality's progress. Tom -------------------------------------------------------------- Reply to this message by going to Community [http://community.jboss.org/message/627698#627698] Start a new discussion in JBoss Transactions Development at Community [http://community.jboss.org/choose-container!input.jspa?contentType=1&...]

Jason Greene [http://community.jboss.org/people/jason.greene] created the discussion "Re: Remoting Transport Transaction Inflow Design Discussion" To view the discussion, visit: http://community.jboss.org/message/627831#627831 -------------------------------------------------------------- Just to clarify my position is 1. We need a decent stop-gap for EAP 6.0, whatever that may be 2. We need the real solution in EAP 6.1 The fork comment wasn't emotive, it was a practical reality. If the TX project was not going to do a 4 release that fixed the problem, then soemone else would have to. It's essentially the same way we handle a thirdparty project (e.g. apache) that does not share our schedule. -------------------------------------------------------------- Reply to this message by going to Community [http://community.jboss.org/message/627831#627831] Start a new discussion in JBoss Transactions Development at Community [http://community.jboss.org/choose-container!input.jspa?contentType=1&...]

Tom Jenkinson [http://community.jboss.org/people/tomjenkinson] created the discussion "Re: Remoting Transport Transaction Inflow Design Discussion" To view the discussion, visit: http://community.jboss.org/message/627838#627838 -------------------------------------------------------------- +1 I do believe there has been some misinterpretation in regards the content of the thread. We are very happy to add this feature into 6.1 and work with you to provide an intermediate solution for 6. The main concern we have is terms of the intermediate solution is about ensuring we deliver a quality supportable component (i.e. QE/docs) in the timescale available. Perhaps now though is the time to move on to the more exciting part, designing this feature work and seeing what can realistically make it into 6 vs 6.1? -------------------------------------------------------------- Reply to this message by going to Community [http://community.jboss.org/message/627838#627838] Start a new discussion in JBoss Transactions Development at Community [http://community.jboss.org/choose-container!input.jspa?contentType=1&...]

Tom Jenkinson [http://community.jboss.org/people/tomjenkinson] created the discussion "Re: Remoting Transport Transaction Inflow Design Discussion" To view the discussion, visit: http://community.jboss.org/message/628210#628210 -------------------------------------------------------------- Hi, Just to keep you all in the loop I have now created https://issues.jboss.org/browse/JBTM-895 https://issues.jboss.org/browse/JBTM-895 to track this work more formally. I have raised it as Blocker on 4.15.x and have personally assigned it to myself. By all means we can keep discussions going in here but at a suitable point it would be worth (perhaps when we have more than  "more details coming soon" in the description of the Jira) we can transition discussion to this Jira. Jason and David, I have added you as watchers on this Jira, feel free to become Voters too on it but as you know, we are working on this already :) I will spend the rest of today, this weekend and Monday fleshing out the description of the Jira with what we have so far. Tom -------------------------------------------------------------- Reply to this message by going to Community [http://community.jboss.org/message/628210#628210] Start a new discussion in JBoss Transactions Development at Community [http://community.jboss.org/choose-container!input.jspa?contentType=1&...]

Mark Little [http://community.jboss.org/people/marklittle] created the discussion "Re: Remoting Transport Transaction Inflow Design Discussion" To view the discussion, visit: http://community.jboss.org/message/629338#629338 -------------------------------------------------------------- And created design discussion entries in the forums too please :) -------------------------------------------------------------- Reply to this message by going to Community [http://community.jboss.org/message/629338#629338] Start a new discussion in JBoss Transactions Development at Community [http://community.jboss.org/choose-container!input.jspa?contentType=1&...]

Tom Jenkinson [http://community.jboss.org/people/tomjenkinson] created the discussion "Re: Remoting Transport Transaction Inflow Design Discussion" To view the discussion, visit: http://community.jboss.org/message/629544#629544 -------------------------------------------------------------- How does this look as an interface to drive the required behaviour? I would assume you would acquire a singleton instance of this through a getter on the impl, e.g. JTADistributionManager manager = JTADistributionsManagerImpl.getJTADistributionManager() public interface JTADistributionManager {      // TRANSACTION ASSOCIATION MANAGEMENT      void start(Xid xid, long timeout) // This should be the absolute time you want the transaction to timeout      void resume(Xid xid) // TO BE CALLED WHEN YOU WANT TO RESUME THE TRANSACTION AT THIS NODE      void suspend(Xid xid)  // TO BE CALLED WHEN YOU WANT TO SUSPEND THE TRANSACTION AT THIS NODE      /// TRANSACTION LIFECYCLE MANAGEMENT      int prepare(Xid xid) throws XAException;      void commit(Xid xid) throws XAException;      void rollback(Xid xid) throws XAException;      void forget(Xid xid)      Xid[] recover(int flag)      // SYNCHRONIZATION CONTROL      void beforeCompletion(Xid xid);      void afterCompletion(Xid xid, int status); } NOTES: Regarding the timeout, I figured the transport would converted this to an absolute time? I can only really see it working with an absolute time and clocks in sync, then when we add it to the reaper we can calculate the remaining time. I am interested to know in the original question some of Davids concerns regarding timeout. Assumptions: 1. Every time remoting invokes a remote application server it must do the following two things: a. If not already done so, create a proxy Synchronization object and register it with the transaction - distributed JTA interposed synchronizations are not supported in the initial version. This will invoke the beforeCompletion and afterCompletion methods on the remote JTADistributionManager b. f not already done so, create a proxy XAResource object and register it with the transaction, this is responsible for sending the subordinate transaction managers transaction lifecycle messages prepare, commit, rollback, forget, recover, setTransactionTimeout c. On the remote side, it needs to call either start/resume dependent on whether remoting detects it has sent the transaction to this server before d. After remoting detects the interaction is complete, it can call end on the JTADistributionManager 2. As we have no way to communicate back up the tree, I have assumed that only the originating client can initiate completion of a transaction, this is consistent with the JTA spec 3.2.2 but will need documenting 3. In the case of failure afterCompletion may not be called As yet unresolved is how to modify the Xid correctly in order that local recovery managers detect the indoubt transactions, watch this space, but this is being tackled, David mentioned an approach on the Synchronizations thread that may assist: http://community.jboss.org/thread/172869?tstart=0 http://community.jboss.org/thread/172869?tstart=0 If you are happy with the proposed interface I will update: https://issues.jboss.org/browse/JBTM-895 https://issues.jboss.org/browse/JBTM-895 -------------------------------------------------------------- Reply to this message by going to Community [http://community.jboss.org/message/629544#629544] Start a new discussion in JBoss Transactions Development at Community [http://community.jboss.org/choose-container!input.jspa?contentType=1&...]

Mark Little [http://community.jboss.org/people/marklittle] created the discussion "Re: Remoting Transport Transaction Inflow Design Discussion" To view the discussion, visit: http://community.jboss.org/message/630236#630236 -------------------------------------------------------------- For distributed timeouts check out what we do with the JTS. -------------------------------------------------------------- Reply to this message by going to Community [http://community.jboss.org/message/630236#630236] Start a new discussion in JBoss Transactions Development at Community [http://community.jboss.org/choose-container!input.jspa?contentType=1&...]

Tom Jenkinson [http://community.jboss.org/people/tomjenkinson] created the discussion "Re: Remoting Transport Transaction Inflow Design Discussion" To view the discussion, visit: http://community.jboss.org/message/642551#642551 -------------------------------------------------------------- Just to confirm, the work for the transactions team side of this is now complete (as part of https://issues.jboss.org/browse/JBTM-895 https://issues.jboss.org/browse/JBTM-895 and friends). If you look at the example I have mentioned before ( https://svn.jboss.org/repos/labs/labs/jbosstm/branches/JBOSSTS_4_16/atsin... https://svn.jboss.org/repos/labs/labs/jbosstm/branches/JBOSSTS_4_16/atsin...) you will see how to achieve use the APIs we have exposed to perform this work within the transport tier. Of particular note are: 1. The transport is responsible for providing a *Serializable* XAResource that is able to proxy calls to remote servers. 2. Per transaction, only one link should be built to each subordinate, this is relatively trivial to ensure and the example provides a suggestion based on registering the resource after returning from the remote server where it is easy to detect an existing subordinate atomic action, note this applies the transaction as a global notion, i.e. do not create a subordinate transaction at the originating server even when the transaction flows back in to the server, the example shows one way how to guard against this. 3. The transport should have no need to persist additional details of transaction state if the pattern illustrated in the example is followed. 4. The transport is *not* responsible for maintaing a list of subordinate identifiers, that requirement is removed and this is managed by the transaction manager. 5. Per subordinate transaction a proxy synchronization and proxy xa resource will need to be registered. 6. As requested, the node names are able to be stored as Strings. The strings *must* be able to be String::getBytes()'d into a byte array no longer than 32 bytes. 7. After you return from a remote server you are suggested to check the state of the transaction returned in order to prevent needless propagation of a transaction that is set in rollback only state. 8. Transaction timeout is shown how to be handled in the example and as discussed in a different thread, it should not need "padding" 9. The transport is responsible for providing an org.jboss.tm.XAResourceRecovery to detect orphan subordinate transactions, this is also illustrated in the example, this *must* be the last registered XAResourceRecovery. The transports implementation of this class must use a transport specific implementation to talk to any possible server that a transaction may have flowed to in order to discover potential orphan transactions. This is because the remote side may have a prepared a transaction and then crashed before the local side has prepared therefore we have no other options available to us. In the example I hard code all the remote server nodes before hand, an alternative is to build this list up incrementally at each node as the transport detects a new remote server is being communicated with. 10. The call to com.arjuna.ats.jta.common.jtaPropertyManager.getJTAEnvironmentBean().setXaResourceOrphanFilterClassNames() should add a filter for: "com.arjuna.ats.internal.jta.recovery.arjunacore.SubordinateJTAXAResourceOrphanFilter" When reading the example, one approach would be to look at: com.arjuna.jta.distributed.example.server.impl.ProxyXAResource as the local interceptor for an EJB (say) and consider its counterpart remote interceptor to be something along the lines of: com.arjuna.jta.distributed.example.server.impl.RemoteServerImpl. The rest of the code should fall into place when the responsibilities of these classes are understood, alongside the addtional work mentioned in actually bringing the server up in a mode that can recover a transports ProxyXAResource-esque class. Any problems please do let us know, it sounds a lot more complicated that it is in reality! The key part is getting the proxy registered and recoverable which hopefully the transport demonstrates in a relatively straightforward manner - not the use of classloaders in the example is to allow multiple transaction managers to run in the same VM - this is for testing and is not a requirement of this feature and can be largely ignored. -------------------------------------------------------------- Reply to this message by going to Community [http://community.jboss.org/message/642551#642551] Start a new discussion in JBoss Transactions Development at Community [http://community.jboss.org/choose-container!input.jspa?contentType=1&...]