Re: [jboss-as7-dev] Examples of heuristic outcome and other transaction errors

On 07/11/2011 12:31 AM, Misty Stanley-Jones wrote: hmm yes, forcing heuristic outcomes is not easy - the system does everything in its power to avoid them, so making one occur is hard work. However it's a common request from QE, GSS and the occasional customer, so here is the stock answer: 1) run under high load and disable, either in software or hardware, one or more of the resource managers e.g. physically pull out the network cable connecting the app server to the resource manager. This is a probabilistic outcome - if the load is high enough chances are you'll hit at least one transaction at the right point in the lifecycle to cause a heuristic. Finding it in the huge server log files may take a while though :-) 2) Run a single transaction, hold it on the debugger at the critical point, disable the RM and allow execution to continue. More elegant, but requires some understanding of the source code (hint: BasicAction.prepare) 3) Write a dummy XAResource for enlistment into the transaction and have it fail at an appropriate point e.g. in the commit call. 4) Use Byteman to instrument an existing XAResource to fail at an appropriate point. These approaches all have one thing in common: they are intended for software engineers. Unless you have a tame developer to hand, your best bet will be to find a pre-existing example. The TS testsuite has several, although not in the most user-friendly form. There was also some work done for the anti-FUD video which is probably still hanging around somewhere. As for troubleshooting advise: Heuristics typically fall into two groups - those caused by transient failures in the environment and those caused by coding errors. The former are more common in production or stress tests, whilst the latter are normally weeded out at development time. Heuristics caused by transient failures rarely need much in the way of root cause analysis - the network break / db server outage / power loss / etc is normally readily apparent. The tricky bit is resolving by hand any transactions that are in a heuristically completed state. The TS system only automatically resolves pending transactions during recovery, not heuristically completed ones. Cleaning those up can require identifying which RMs were involved, examining state in the transaction manager and RMs and manually forcing log cleanup and data reconciliation in one or more of them. It's not easy and the exact steps depend on the specific RMs involved (each has its own management tooling) and the data content of the transaction branch in the RMs. I'm in the process of adding additional logging output to assist with this type of thing, but from the point of view of the AS7/EAP6 user the biggest problem right now is we're missing an integrated tx objectstore browser frontend :-( Heuristics caused by misbehaving (i.e. not spec compliant) RMs are less common but do occur occasionally. Unless the user is also the RM author, troubleshooting these is normally a job for GSS and the TS devs. The root cause is normally some disagreement or misunderstanding in the interpretation of an arcane point in the spec. In practical terms the cleanup is simpler, as in dev environments simply deleting all the relevant logs records in the RM and app server will reset things - data reconciliation is rarely a significant concern here. As for general (non-heuristic) error handling in transactions, the key cases are: - transaction timed out in the background and the business logic thread did not notice. This frequently manifests in hibernate suddenly being unable to get a db connection for lazy loading. begin a new hibernate session. lengthen the timeout, restructure your code or tune the environment if it's happening frequently. - transaction already running on a thread, normally because thread pooling is used and something forgot to disassociate an old transaction before releasing the thread back to the pool. Use finally blocks for cleanup and be aware that TransactionManager.[commit|rollback] do disassociation but Transaction.[commit|rollback] don't. Yes I'm looking at you Mr. Spring Framework. - recovery failed due to non-serializable resource. Usually caused by failing to configure the recovery plugin properly and hopefully a thing of the past now that is at last done automatically by the server :-) - the system disallowed enlistment of a second local resource. This is just a problem of user education - the difference between local-ds and xa-ds is not well understood. I think those cover a majority of the support load, but the GSS guys may have others. Jonathan. -- Registered Address: Red Hat UK Ltd, Amberley Place, 107-111 Peascod Street, Windsor, Berkshire, SI4 1TE, United Kingdom. Registered in UK and Wales under Company Registration No. 3798903 Directors: Michael Cunningham (USA), Charlie Peters (USA), Matt Parsons (USA) and Brendan Lane (Ireland).