Forwarding to JBossCache dev. Manik et all, Thoughts on this? Cheers, -------- Original Message -------- Subject: [Fwd: New case comment notification. Case Number 00017677] Date: Thu, 16 Aug 2007 11:46:18 +0100 From: Jonathan Halliday <jonathan.halliday(a)redhat.com&gt; To: manik.surtani(a)jboss.com CC: support-transactions(a)jboss.com, support-as(a)jboss.com Hi Manik Although this support issue is ultimately an Oracle problem, it does expose a weakness in the JBoss Cache. Where an XA transaction has a heuristic outcome, the status reported to the afterCompletion synchronization is STATUS_UNKNOWN (5), since JTA does not handle heuristics. In this case, the cache Synchronization fails to rollback the cache tx. This leaves the app server in a bad (i.e. locked) state even though the XA transaction has been resolved. At first glance it looks like the afterCompletion synchronization should treat a STATUS_UNKNOWN as a cause for rollback as well as logging the error as they currently do. Unfortunately the fix may be more complex than that, since the db may have committed rather than rolled back, meaning the cache would be out of sync with the db if you rolled it back. What you really need is proper XA resources rather than synchronizations for the cache, but that is a much bigger job. [ EAP 4.2, i.e. cache 1.4.1.SP3] 2007-08-15 16:24:13,734,ERROR,,,,[OrderedSynchronizationHandler],JMS SessionPool Worker-0,failed calling afterCompletion() on TxInte rceptor.LocalSynchronizationHandler (gtx=GlobalTransaction:<10.25.98.165:3048>:2, tx=TransactionImple < ac, BasicAction: a1962a5:be3: 46c36d0d:da status: ActionStatus.COMMITTED >)^M 11806 java.lang.IllegalStateException: illegal status: 5^M 11807 at org.jboss.cache.interceptors.TxInterceptor $RemoteSynchronizationHandler.afterCompletion(TxInterceptor.java: 1088)^M 11808 at org.jboss.cache.interceptors.TxInterceptor $LocalSynchronizationHandler.afterCompletion(TxInterceptor.java: 1175)^M 11809 at org.jboss.cache.interceptors.OrderedSynchronizationHandler.afterComple tion(OrderedSynchronizationHandler.java:83)^M 11810 at com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple. afterCompletion(SynchronizationImple.java:136)^M 11811 at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.afterCompletion( TwoPhaseCoordinator.java:340)^M Regards Jonathan. -------- Original Message -------- Subject: New case comment notification. Case Number 00017677 Date: Thu, 16 Aug 2007 10:31:36 +0000 (GMT) From: Jonathan Halliday <jhallida(a)redhat.com&gt; To: support-as(a)jboss.com <support-as(a)jboss.com&gt; Jonathan Halliday has added a comment to case 00017677 : "JBoss 4.2 and OracleXA Error - XAException.XAER_RMERR". Please read the comment below and then click on the link to respond appropriately. Comment: The logs show that Oracle is throwing an error during an XA commit. Since it's in the second phase of the 2PC, this leads to a heuristic outcome to the transaction. Unfortunately JBoss cache is not handling this well, leading it to keep locks on some data. Subsequent errors cascade from this. You need to take up the matter with DataDirect and Oracle to determine why it's failing on the commit. spy(JMS SessionPool Worker-0)>> XAResource[5].commit(Xid xid, boolean onePhase) spy(JMS SessionPool Worker-0)>> xid = 131075-312D613139363261353A6265-613139363261353A626533 spy(JMS SessionPool Worker-0)>> onePhase = false JJH spy(JMS SessionPool Worker-0)>> *** XAException javax.transaction.xa.XAException: [DataDirect][Oracle JDBC Driver]Oracle XA Error Occurred. Native Error: 1 -3=XAER_RMERR *** javax.transaction.xa.XAException: [DataDirect][Oracle JDBC Driver]Oracle XA Error Occurred. Native Error: 1 at com.ddtek.jdbcx.oracle.OracleImplXAResource.checkError(Unknown Source) at com.ddtek.jdbcx.oracle.OracleImplXAResource.commit(Unknown Source) at com.ddtek.jdbcx.base.BaseXAResource.commit(Unknown Source) at com.ddtek.jdbcspy.SpyXAResource.commit(Unknown Source) at org.jboss.resource.adapter.jdbc.xa.XAManagedConnection.commit (XAManagedConnection.java:175) at org.jboss.resource.connectionmanager.xa.JcaXAResourceWrapper.commit (JcaXAResourceWrapper.java:53) at com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord.topL evelCommit(XAResourceRecord.java:487) https://na1.salesforce.com/50030000003QAVl -- ------------------------------------------------------------ Registered Address: Red Hat UK Ltd, Amberley Place, 107-111 Peascod Street, Windsor, Berkshire, SI4 1TE, United Kingdom. Registered in UK and Wales under Company Registration No. 3798903 Directors: Michael Cunningham (USA), Charlie Peters (USA) and David Owens (Ireland) -- Galder Zamarreño Sr. Software Maintenance Engineer JBoss, a division of Red Hat _______________________________________________ jbosscache-dev mailing list jbosscache-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/jbosscache-dev

Manik Surtani wrote: > Sorry about the slow response on this. > While making JBC an XA resource is on the roadmap, it isn't > something scheduled for any imminent release [1], as this is a > pretty big change for the JBC codebase (and more importantly, a > big feature change which would involve a lot of compatibility > testing). > For the time being, treating STATUS_UNKNOWN as a rollback would > help, but only provided the database (and any other resources > participating in the tx) has rolled back as well. Jonathan, > correct me if I am wrong, but there is no way to know this unless > we are an XA resource as well, am I right? > So is it reasonable to treat a STATUS_UNKNOWN as a rollback, and > document this limitation, until JBCACHE-70? As far as I understood, there's two issues here: - the fact that we through an ISE that messes the TM. - the fact that we don't clear any locks being held, which leads to future lock timeout exceptions.

> Cheers > Manik > [1] http://jira.jboss.com/jira/browse/JBCACHE-70 > On 20 Aug 2007, at 13:49, Galder Zamarreno wrote: >> Forwarding to JBossCache dev. >> >> Manik et all, >> >> Thoughts on this? >> >> Cheers, >> >> -------- Original Message -------- >> Subject: [Fwd: New case comment notification. Case Number 00017677] >> Date: Thu, 16 Aug 2007 11:46:18 +0100 >> From: Jonathan Halliday <jonathan.halliday(a)redhat.com&gt; >> To: manik.surtani(a)jboss.com >> CC: support-transactions(a)jboss.com, support-as(a)jboss.com >> >> >> Hi Manik >> >> Although this support issue is ultimately an Oracle problem, >> it does expose a weakness in the JBoss Cache. Where an XA >> transaction has a heuristic outcome, the status reported to >> the afterCompletion synchronization is STATUS_UNKNOWN (5), >> since JTA does not handle heuristics. In this case, the >> cache Synchronization fails to rollback the cache tx. This >> leaves the app server in a bad (i.e. locked) state even >> though the XA transaction has been resolved. >> >> At first glance it looks like the afterCompletion >> synchronization should treat a STATUS_UNKNOWN as a cause for >> rollback as well as logging the error as they currently do. >> Unfortunately the fix may be more complex than that, since >> the db may have committed rather than rolled back, meaning >> the cache would be out of sync with the db if you rolled it >> back. What you really need is proper XA resources rather >> than synchronizations for the cache, but that is a much >> bigger job. >> >> [ EAP 4.2, i.e. cache 1.4.1.SP3] >> 2007-08-15 >> 16:24:13,734,ERROR,,,,[OrderedSynchronizationHandler],JMS >> SessionPool Worker-0,failed calling afterCompletion() on >> TxInte >> rceptor.LocalSynchronizationHandler >> (gtx=GlobalTransaction:<10.25.98.165:3048>:2, >> tx=TransactionImple < ac, BasicAction: a1962a5:be3: >> 46c36d0d:da status: ActionStatus.COMMITTED >)^M >> 11806 java.lang.IllegalStateException: illegal status: 5^M >> 11807 at >> org.jboss.cache.interceptors.TxInterceptor >> $RemoteSynchronizationHandler.afterCompletion(TxInterceptor.java: >> 1088)^M >> 11808 at >> org.jboss.cache.interceptors.TxInterceptor >> $LocalSynchronizationHandler.afterCompletion(TxInterceptor.java: >> 1175)^M >> 11809 at >> org.jboss.cache.interceptors.OrderedSynchronizationHandler.afterComp >> letion(OrderedSynchronizationHandler.java:83)^M >> 11810 at >> com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImpl >> e.afterCompletion(SynchronizationImple.java:136)^M >> 11811 at >> com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.afterCompletio >> n(TwoPhaseCoordinator.java:340)^M >> >> >> Regards >> >> Jonathan. >> >> >> >> >> -------- Original Message -------- >> Subject: New case comment notification. Case Number 00017677 >> Date: Thu, 16 Aug 2007 10:31:36 +0000 (GMT) >> From: Jonathan Halliday <jhallida(a)redhat.com&gt; >> To: support-as(a)jboss.com <support-as(a)jboss.com&gt; >> >> Jonathan Halliday has added a comment to case 00017677 : >> "JBoss 4.2 and OracleXA Error - XAException.XAER_RMERR". >> Please read the comment below and then click on the link to >> respond appropriately. >> >> Comment: >> The logs show that Oracle is throwing an error during an XA >> commit. Since it's in the second phase of the 2PC, this >> leads to a heuristic outcome to the transaction. >> Unfortunately JBoss cache is not handling this well, leading >> it to keep locks on some data. Subsequent errors cascade >> from this. You need to take up the matter with DataDirect >> and Oracle to determine why it's failing on the commit. >> >> spy(JMS SessionPool Worker-0)>> XAResource[5].commit(Xid >> xid, boolean onePhase) >> >> spy(JMS SessionPool Worker-0)>> xid = >> 131075-312D613139363261353A6265-613139363261353A626533 >> >> spy(JMS SessionPool Worker-0)>> onePhase = false >> >> JJH spy(JMS SessionPool Worker-0)>> *** XAException >> javax.transaction.xa.XAException: [DataDirect][Oracle JDBC >> Driver]Oracle XA Error Occurred. Native Error: 1 >> -3=XAER_RMERR *** >> >> javax.transaction.xa.XAException: [DataDirect][Oracle JDBC >> Driver]Oracle XA Error Occurred. Native Error: 1 >> >> at >> com.ddtek.jdbcx.oracle.OracleImplXAResource.checkError(Unknown >> Source) >> >> at >> com.ddtek.jdbcx.oracle.OracleImplXAResource.commit(Unknown >> Source) >> >> at com.ddtek.jdbcx.base.BaseXAResource.commit(Unknown Source) >> >> at com.ddtek.jdbcspy.SpyXAResource.commit(Unknown Source) >> >> at >> org.jboss.resource.adapter.jdbc.xa.XAManagedConnection.commit >> (XAManagedConnection.java:175) >> >> at >> org.jboss.resource.connectionmanager.xa.JcaXAResourceWrapper.commit( >> JcaXAResourceWrapper.java:53) >> >> at >> com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord.to >> pLevelCommit(XAResourceRecord.java:487) >> >> https://na1.salesforce.com/50030000003QAVl >> >> >> >> >> -- >> ------------------------------------------------------------ >> Registered Address: Red Hat UK Ltd, Amberley Place, 107-111 >> Peascod Street, Windsor, Berkshire, SI4 1TE, United Kingdom. >> Registered in UK and Wales under Company Registration No. >> 3798903 Directors: Michael Cunningham (USA), Charlie Peters >> (USA) and David Owens (Ireland) >> >> -- >> Galder Zamarreño >> Sr. Software Maintenance Engineer >> JBoss, a division of Red Hat >> _______________________________________________ >> jbosscache-dev mailing list >> jbosscache-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/jbosscache-dev > -- > Manik Surtani > Lead, JBoss Cache > JBoss, a division of Red Hat -- Galder Zamarreño Sr. Software Maintenance Engineer JBoss, a division of Red Hat