Receiving "Trying to change Tx in enlist!" errors in multi-threaded modules. ----------------------------------------------------------------------------- Key: JBAS-7419 URL: https://jira.jboss.org/jira/browse/JBAS-7419 Project: JBoss Application Server Issue Type: Bug Security Level: Public(Everyone can see) Components: Transaction Manager (JBossTM) Affects Versions: JBossAS-4.0.2 Final Environment: Java version: 1.5.0,IBM Corporation, Java VM: IBM J9 VM 2.3,IBM Corporation, OS-System: AIX 5.3,ppc64 Reporter: Shankar Shantharaju Assignee: Adrian Brock Priority: Blocker Receiving "Trying to change Tx in enlist!" errors in multi-threaded modules. There are three different scenarios in which this error is generated. This error occurs in random and there is no definited pattern. A similar issue was reported and fixed as per the link: http://www.mail-archive.com/jboss-development@lists.sourceforge.net/msg30... in an older version of JBoss AS. Any help as early as possbile is hhighly appreciated. 1. Message Driven Bean: On posting messages to Queue which being associated with a Message Driven bean, the following error is being generated. 21:42:06,084 WARN [org.jboss.resource.connectionmanager.TxConnectionManager] in Enlisting tx, trying to change tx. illegal s tate: old: TransactionImpl:XidImpl[FormatId=257, GlobalId=DBPHQRPLQ01/112154, BranchQual=, localId=112154], new: TransactionI mpl:XidImpl[FormatId=257, GlobalId=DBPHQRPLQ01/112155, BranchQual=, localId=112155], cel: org.jboss.resource.connectionmanage r.TxConnectionManager$TxConnectionEventListener@175b175b[state=NORMAL mc=org.jboss.resource.adapter.jdbc.xa.XAManagedConnecti on@16c916c9 handles=0 lastUse=1257133326083 permit=true trackByTx=true mcp=org.jboss.resource.connectionmanager.JBossManagedC onnectionPool$OnePool@3cbc3cbc context=org.jboss.resource.connectionmanager.InternalManagedConnectionPool@33373337] 21:42:06,085 ERROR [org.jboss.ejb.plugins.jms.JMSContainerInvoker] Exception in JMSCI message listener org.jboss.resource.JBossResourceException: Unchecked throwable in managedConnectionReconnected(); - nested throwable: (java.l ang.IllegalStateException: Trying to change Tx in enlist!) at org.jboss.resource.connectionmanager.BaseConnectionManager2.rethrowAsResourceException(BaseConnectionManager2.java :101) at org.jboss.resource.connectionmanager.BaseConnectionManager2.reconnectManagedConnection(BaseConnectionManager2.java :588) at org.jboss.resource.connectionmanager.BaseConnectionManager2.reconnect(BaseConnectionManager2.java:508) at org.jboss.resource.connectionmanager.CachedConnectionManager.reconnect(CachedConnectionManager.java:421) at org.jboss.resource.connectionmanager.CachedConnectionManager.pushMetaAwareObject(CachedConnectionManager.java:261) at org.jboss.resource.connectionmanager.CachedConnectionInterceptor.invoke(CachedConnectionInterceptor.java:182) at org.jboss.ejb.plugins.MessageDrivenInstanceInterceptor.invoke(MessageDrivenInstanceInterceptor.java:87) at org.jboss.ejb.plugins.CallValidationInterceptor.invoke(CallValidationInterceptor.java:48) at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:105) at org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:335) at org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java:166) at org.jboss.ejb.plugins.RunAsSecurityInterceptor.invoke(RunAsSecurityInterceptor.java:94) at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:192) at org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java:122) at org.jboss.ejb.MessageDrivenContainer.internalInvoke(MessageDrivenContainer.java:389) at org.jboss.ejb.Container.invoke(Container.java:873) at org.jboss.ejb.plugins.jms.JMSContainerInvoker.invoke(JMSContainerInvoker.java:1090) at org.jboss.ejb.plugins.jms.JMSContainerInvoker$MessageListenerImpl.onMessage(JMSContainerInvoker.java:1392) at org.jboss.jms.asf.StdServerSession.onMessage(StdServerSession.java:256) at org.jboss.mq.SpyMessageConsumer.sessionConsumerProcessMessage(SpyMessageConsumer.java:904) at org.jboss.mq.SpyMessageConsumer.addMessage(SpyMessageConsumer.java:160) at org.jboss.mq.SpySession.run(SpySession.java:333) at org.jboss.jms.asf.StdServerSession.run(StdServerSession.java:180) at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:748) at java.lang.Thread.run(Thread.java:810) Caused by: java.lang.IllegalStateException: Trying to change Tx in enlist! at org.jboss.resource.connectionmanager.TxConnectionManager$TxConnectionEventListener.enlist(TxConnectionManager.java :417) at org.jboss.resource.connectionmanager.TxConnectionManager.managedConnectionReconnected(TxConnectionManager.java:322 ) at org.jboss.resource.connectionmanager.BaseConnectionManager2.reconnectManagedConnection(BaseConnectionManager2.java :583) ... 23 more 2. Threads that get created inside the application server and start their own independent transactions: org.springframework.jdbc.UncategorizedSQLException: Hibernate operation: Cannot open connection; uncategorized SQLException f or SQL [???]; SQL state [null]; error code [0]; Unchecked throwable in managedConnectionReconnected(); - nested throwable: (j ava.lang.IllegalStateException: Trying to change Tx in enlist!); - nested throwable: (org.jboss.resource.JBossResourceExcepti on: Unchecked throwable in managedConnectionReconnected(); - nested throwable: (java.lang.IllegalStateException: Trying to ch ange Tx in enlist!)); nested exception is org.jboss.util.NestedSQLException: Unchecked throwable in managedConnectionReconnec ted(); - nested throwable: (java.lang.IllegalStateException: Trying to change Tx in enlist!); - nested throwable: (org.jboss. resource.JBossResourceException: Unchecked throwable in managedConnectionReconnected(); - nested throwable: (java.lang.Illega lStateException: Trying to change Tx in enlist!)) Caused by: org.jboss.util.NestedSQLException: Unchecked throwable in managedConnectionReconnected(); - nested throwable: (java.lang.Ille galStateException: Trying to change Tx in enlist!); - nested throwable: (org.jboss.resource.JBossResourceException: Unchecked throwable in managedConnectionReconnected(); - nested throwable: (java.lang.IllegalStateException: Trying to change Tx in en list!)) at org.jboss.resource.adapter.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:107) at org.springframework.orm.hibernate3.LocalDataSourceConnectionProvider.getConnection(LocalDataSourceConnectionProvid er.java:81) at org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:423) at org.hibernate.jdbc.ConnectionManager.getConnection(ConnectionManager.java:144) at org.hibernate.jdbc.AbstractBatcher.prepareQueryStatement(AbstractBatcher.java:139) at org.hibernate.loader.Loader.prepareQueryStatement(Loader.java:1557) at org.hibernate.loader.Loader.doQuery(Loader.java:683) at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:246) at org.hibernate.loader.Loader.doList(Loader.java:2223) at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2114) at org.hibernate.loader.Loader.list(Loader.java:2109) at org.hibernate.loader.criteria.CriteriaLoader.list(CriteriaLoader.java:94) at org.hibernate.impl.SessionImpl.list(SessionImpl.java:1569) at org.hibernate.impl.CriteriaImpl.list(CriteriaImpl.java:283) at com.manh.df.base.persistence.dao.hibernate.HibernateListCriteriaHelper$1.doInHibernate(HibernateListCriteriaHelper .java:222) at org.springframework.orm.hibernate3.HibernateTemplate.execute(HibernateTemplate.java:372) at com.manh.df.base.persistence.dao.hibernate.HibernateListCriteriaHelper.getListChunk(HibernateListCriteriaHelper.ja va:205) at com.manh.df.base.persistence.dao.hibernate.HibernateListCriteriaHelper.getListChunk(HibernateListCriteriaHelper.ja va:66) at com.manh.df.base.persistence.dao.hibernate.HibernateProductLocationDao.getProductLocationList(HibernateProductLoca tionDao.java:195) at com.manh.df.base.service.impl.ProductLocationServiceImpl.getProductLocationList(ProductLocationServiceImpl.java:14 5) at sun.reflect.GeneratedMethodAccessor1323.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:618) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:296) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:177) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:144) at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java: 50) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:166) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204) at $Proxy640.getProductLocationList(Unknown Source) at com.manh.df.base.service.impl.AutoAssociationBatchProcessor.getProductLocationList(AutoAssociationBatchProcessor.j ava:1025) at com.manh.df.base.service.impl.AutoAssociationBatchProcessor.associateProductLocationList(AutoAssociationBatchProce ssor.java:249) at com.manh.df.base.service.impl.AutoAssociationBatchProcessor.processRequest(AutoAssociationBatchProcessor.java:136) at com.manh.df.base.async.AsynchronousRequestSupport$1.run(AsynchronousRequestSupport.java:67) at java.security.AccessController.doPrivileged(AccessController.java:219) at javax.security.auth.Subject.doAs(Subject.java:495) at com.manh.df.base.util.PrivilegedActionRunner.doPrivilegedAction(PrivilegedActionRunner.java:84) at com.manh.df.base.util.PrivilegedActionRunner.doPrivilegedAction(PrivilegedActionRunner.java:45) at com.manh.df.base.async.AsynchronousRequestSupport.run(AsynchronousRequestSupport.java:52) Caused by: org.jboss.resource.JBossResourceException: Unchecked throwable in managedConnectionReconnected(); - nested throwable: (java.l ang.IllegalStateException: Trying to change Tx in enlist!) at org.jboss.resource.connectionmanager.BaseConnectionManager2.rethrowAsResourceException(BaseConnectionManager2.java :101) at org.jboss.resource.connectionmanager.BaseConnectionManager2.reconnectManagedConnection(BaseConnectionManager2.java :588) at org.jboss.resource.connectionmanager.BaseConnectionManager2.allocateConnection(BaseConnectionManager2.java:450) at org.jboss.resource.connectionmanager.BaseConnectionManager2$ConnectionManagerProxy.allocateConnection(BaseConnecti onManager2.java:874) at org.jboss.resource.adapter.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:103) ... 38 more Caused by: java.lang.IllegalStateException: Trying to change Tx in enlist! at org.jboss.resource.connectionmanager.TxConnectionManager$TxConnectionEventListener.enlist(TxConnectionManager.java :417) at org.jboss.resource.connectionmanager.TxConnectionManager.managedConnectionReconnected(TxConnectionManager.java:322 ) at org.jboss.resource.connectionmanager.BaseConnectionManager2.reconnectManagedConnection(BaseConnectionManager2.java :583) ... 41 more 3. Errors in External multi threaded client module which makes remote calls to EJB services: org.jboss.resource.JBossResourceException: Unchecked throwable in managedConnectionReconnected(); - nested throwable: (java.lang.IllegalStateException: Trying to change Tx in enlist!) at org.jboss.resource.connectionmanager.BaseConnectionManager2.rethrowAsResourceException(BaseConnectionManager2.java:101) at org.jboss.resource.connectionmanager.BaseConnectionManager2.reconnectManagedConnection(BaseConnectionManager2.java:588) at org.jboss.resource.connectionmanager.BaseConnectionManager2.reconnect(BaseConnectionManager2.java:508) at org.jboss.resource.connectionmanager.CachedConnectionManager.reconnect(CachedConnectionManager.java:421) at org.jboss.resource.connectionmanager.CachedConnectionManager.pushMetaAwareObject(CachedConnectionManager.java:261) at org.jboss.resource.connectionmanager.CachedConnectionInterceptor.invokeHome(CachedConnectionInterceptor.java:209) at org.jboss.ejb.plugins.StatelessSessionInstanceInterceptor.invokeHome(StatelessSessionInstanceInterceptor.java:81) at org.jboss.ejb.plugins.AbstractInterceptor.invokeHome(AbstractInterceptor.java:90) at org.jboss.ejb.plugins.CallValidationInterceptor.invokeHome(CallValidationInterceptor.java:41) at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:109) at org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:335) at org.jboss.ejb.plugins.TxInterceptorCMT.invokeHome(TxInterceptorCMT.java:146) at org.jboss.ejb.plugins.SecurityInterceptor.invokeHome(SecurityInterceptor.java:116) at org.jboss.ejb.plugins.LogInterceptor.invokeHome(LogInterceptor.java:121) at org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invokeHome(ProxyFactoryFinderInterceptor.java:93) at org.jboss.ejb.SessionContainer.internalInvokeHome(SessionContainer.java:613) at org.jboss.ejb.Container.invoke(Container.java:894) at sun.reflect.GeneratedMethodAccessor281.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:618) at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:141) at org.jboss.mx.server.Invocation.dispatch(Invocation.java:80) at org.jboss.mx.server.Invocation.invoke(Invocation.java:72) at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:249) at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:644) at org.jboss.invocation.jrmp.server.JRMPInvoker$MBeanServerAction.invoke(JRMPInvoker.java:805) at org.jboss.invocation.jrmp.server.JRMPInvoker.invoke(JRMPInvoker.java:406) at sun.reflect.GeneratedMethodAccessor280.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:618) at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:309) at sun.rmi.transport.Transport$1.run(Transport.java:168) at java.security.AccessController.doPrivileged(AccessController.java:279) at sun.rmi.transport.Transport.serviceCall(Transport.java:164) at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:506) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.handleRequest(TCPTransport.java:838) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:912) at java.lang.Thread.run(Thread.java:810) at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:263) at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:239) at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:141) at org.jboss.invocation.jrmp.server.JRMPInvoker_Stub.invoke(Unknown Source) at org.jboss.invocation.jrmp.interfaces.JRMPInvokerProxy.invoke(JRMPInvokerProxy.java:118) at org.jboss.invocation.InvokerInterceptor.invokeInvoker(InvokerInterceptor.java:227) at org.jboss.invocation.InvokerInterceptor.invoke(InvokerInterceptor.java:167) at org.jboss.proxy.TransactionInterceptor.invoke(TransactionInterceptor.java:46) at org.jboss.proxy.SecurityInterceptor.invoke(SecurityInterceptor.java:55) at org.jboss.proxy.ejb.HomeInterceptor.invoke(HomeInterceptor.java:169) at org.jboss.proxy.ClientContainer.invoke(ClientContainer.java:86) ... 22 more Caused by: java.lang.IllegalStateException: Trying to change Tx in enlist! at org.jboss.resource.connectionmanager.TxConnectionManager$TxConnectionEventListener.enlist(TxConnectionManager.java:417) at org.jboss.resource.connectionmanager.TxConnectionManager.managedConnectionReconnected(TxConnectionManager.java:322) at org.jboss.resource.connectionmanager.BaseConnectionManager2.reconnectManagedConnection(BaseConnectionManager2.java:583) at org.jboss.resource.connectionmanager.BaseConnectionManager2.reconnect(BaseConnectionManager2.java:508) at org.jboss.resource.connectionmanager.CachedConnectionManager.reconnect(CachedConnectionManager.java:421) at org.jboss.resource.connectionmanager.CachedConnectionManager.pushMetaAwareObject(CachedConnectionManager.java:261) at org.jboss.resource.connectionmanager.CachedConnectionInterceptor.invokeHome(CachedConnectionInterceptor.java:209) at org.jboss.ejb.plugins.StatelessSessionInstanceInterceptor.invokeHome(StatelessSessionInstanceInterceptor.java:81) at org.jboss.ejb.plugins.AbstractInterceptor.invokeHome(AbstractInterceptor.java:90) at org.jboss.ejb.plugins.CallValidationInterceptor.invokeHome(CallValidationInterceptor.java:41) at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:109) at org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:335) at org.jboss.ejb.plugins.TxInterceptorCMT.invokeHome(TxInterceptorCMT.java:146) at org.jboss.ejb.plugins.SecurityInterceptor.invokeHome(SecurityInterceptor.java:116) at org.jboss.ejb.plugins.LogInterceptor.invokeHome(LogInterceptor.java:121) at org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invokeHome(ProxyFactoryFinderInterceptor.java:93) at org.jboss.ejb.SessionContainer.internalInvokeHome(SessionContainer.java:613) at org.jboss.ejb.Container.invoke(Container.java:894) at sun.reflect.GeneratedMethodAccessor281.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:618) at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:141) at org.jboss.mx.server.Invocation.dispatch(Invocation.java:80) at org.jboss.mx.server.Invocation.invoke(Invocation.java:72) at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:249) at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:644) at org.jboss.invocation.jrmp.server.JRMPInvoker$MBeanServerAction.invoke(JRMPInvoker.java:805) at org.jboss.invocation.jrmp.server.JRMPInvoker.invoke(JRMPInvoker.java:406) at sun.reflect.GeneratedMethodAccessor280.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:618) at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:309) at sun.rmi.transport.Transport$1.run(Transport.java:168) at java.security.AccessController.doPrivileged(AccessController.java:279) at sun.rmi.transport.Transport.serviceCall(Transport.java:164) at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:506) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.handleRequest(TCPTransport.java:838) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:912)