[jboss-jira] [JBoss JIRA] Reopened: (JBAS-4998) App could Dynamically create Connection Pool in 3.0, but crashes in 4.2.1

App could Dynamically create Connection Pool in 3.0, but crashes in 4.2.1 ------------------------------------------------------------------------- Key: JBAS-4998 URL: http://jira.jboss.com/jira/browse/JBAS-4998 Project: JBoss Application Server Issue Type: Bug Security Level: Public(Everyone can see) Components: Transaction Manager, Deployment services Affects Versions: JBossAS-4.2.1.GA, JBossAS-4.2.0.GA Environment: SuSE 10.1 Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_11-b03) Java HotSpot(TM) Client VM (build 1.5.0_11-b03, mixed mode, sharing) JBoss 4.2.1 Reporter: Steve Davidson Attachments: server.log Greetings. Due to the fact that this application has dozen's of different Databases to talk to (later hundreds, then maybe more), other than a 'bootstrap connection pool' to the 'master database, all connection pools have to be created dynamically. The connection information is stored in a master database (the fact that we need a Database to keep track of the connection info for all the databases should say much). In JBoss 3.0 & 3.2.8, it is quite possible to create these connections dynamically. This has been working for some time, actually (starting with JBoss 3.0Beta2). Had to tweak the code a bit when migrating to 3.2.8 a while back, but got it working again quite nicely (still is, actually, in production). Since JBoss 3.x is end of life, we've started migrating to JBoss 4.2.1. The objects that were originally used seem to have been changed. I've updated most of the calls, but at this time, I am getting the following failure; 14:56:39,120 DEBUG [com.security.ejb.UserEJB.initializeState(UserEJB.java:318)] Exception getting Row from Service layer: Unable to obtain Database Connection from the Connection Pool: Could not enlist in transaction on entering meta-aware object!; - nested throwable: (javax.transaction.SystemException: java.lang.Throwable: Unabled to enlist resource, see the previous warnings. tx=TransactionImple < ac, BasicAction: -3f5702ff:38cc:47449b39:2e status: ActionStatus.ABORT_ONLY >); - nested throwable: (org.jboss.resource.JBossResourceException: Could not enlist in transaction on entering meta-aware object!; - nested throwable: (javax.transaction.SystemException: java.lang.Throwable: Unabled to enlist resource, see the previous warnings. tx=TransactionImple < ac, BasicAction: -3f5702ff:38cc:47449b39:2e status: ActionStatus.ABORT_ONLY >)) Error! Unable to obtain Database Connection from the Connection Pool: Could not enlist in transaction on entering meta-aware object!; - nested throwable: (javax.transaction.SystemException: java.lang.Throwable: Unabled to enlist resource, see the previous warnings. tx=TransactionImple < ac, BasicAction: -3f5702ff:38cc:47449b39:2e status: ActionStatus.ABORT_ONLY >); - nested throwable: (org.jboss.resource.JBossResourceException: Could not enlist in transaction on entering meta-aware object!; - nested throwable: (javax.transaction.SystemException: java.lang.Throwable: Unabled to enlist resource, see the previous warnings. tx=TransactionImple < ac, BasicAction: -3f5702ff:38cc:47449b39:2e status: ActionStatus.ABORT_ONLY >))\n\nPlease contact Administrator and relay the following information:\n\nMessage: Could not enlist in transaction on entering meta-aware object!; - nested throwable: (javax.transaction.SystemException: java.lang.Throwable: Unabled to enlist resource, see the previous warnings. tx=TransactionImple < ac, BasicAction: -3f5702ff:38cc:47449b39:2e status: ActionStatus.ABORT_ONLY >); - nested throwable: (org.jboss.resource.JBossResourceException: Could not enlist in transaction on entering meta-aware object!; - nested throwable: (javax.transaction.SystemException: java.lang.Throwable: Unabled to enlist resource, see the previous warnings. tx=TransactionImple < ac, BasicAction: -3f5702ff:38cc:47449b39:2e status: ActionStatus.ABORT_ONLY >))\nSQLState: null\nError Code: 0\n at com.common.ejb.DBConnection.getConnection(DBConnection.java:171) at com.common.ejb.DBConnection.prepareStatement(DBConnection.java:469) at com.common.ejb.DBConnection.executePreparedSQLQuery(DBConnection.java:386) at com.common.ejb.CommonService.getRow(HRXCommonService.java:318) at com.security.ejb.UserEJB.initializeState(UserEJB.java:306) at com.security.ejb.UserEJB.ejbLoad(UserEJB.java:255) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:585) at org.jboss.ejb.plugins.BMPPersistenceManager.loadEntity(BMPPersistenceManager.java:435) at org.jboss.resource.connectionmanager.CachedConnectionInterceptor.loadEntity(CachedConnectionInterceptor.java:252) at org.jboss.ejb.plugins.EntitySynchronizationInterceptor.invoke(EntitySynchronizationInterceptor.java:243) at org.jboss.resource.connectionmanager.CachedConnectionInterceptor.invoke(CachedConnectionInterceptor.java:158) at org.jboss.ejb.plugins.EntityReentranceInterceptor.invoke(EntityReentranceInterceptor.java:126) at org.jboss.ejb.plugins.EntityInstanceInterceptor.invoke(EntityInstanceInterceptor.java:278) at org.jboss.ejb.plugins.EntityLockInterceptor.invoke(EntityLockInterceptor.java:104) at org.jboss.ejb.plugins.EntityCreationInterceptor.invoke(EntityCreationInterceptor.java:76) at org.jboss.ejb.plugins.CallValidationInterceptor.invoke(CallValidationInterceptor.java:63) at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:121) at org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:350) at org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java:181) at org.jboss.ejb.plugins.SecurityInterceptor.invoke(SecurityInterceptor.java:168) at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:205) at org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java:138) at org.jboss.ejb.EntityContainer.internalInvoke(EntityContainer.java:527) at org.jboss.ejb.Container.invoke(Container.java:960) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:585) at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155) at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94) at org.jboss.mx.server.Invocation.invoke(Invocation.java:86) at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264) at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659) at org.jboss.invocation.local.LocalInvoker$MBeanServerAction.invoke(LocalInvoker.java:169) at org.jboss.invocation.local.LocalInvoker.invoke(LocalInvoker.java:118) at org.jboss.invocation.InvokerInterceptor.invokeLocal(InvokerInterceptor.java:209) at org.jboss.invocation.InvokerInterceptor.invoke(InvokerInterceptor.java:195) at org.jboss.proxy.TransactionInterceptor.invoke(TransactionInterceptor.java:61) at org.jboss.proxy.SecurityInterceptor.invoke(SecurityInterceptor.java:70) at org.jboss.proxy.ejb.EntityInterceptor.invoke(EntityInterceptor.java:112) at org.jboss.proxy.ClientContainer.invoke(ClientContainer.java:100) at $Proxy75.logIn(Unknown Source) at com.security.ThinUserProxy.logIn(ThinUserProxy.java:233) at com.security.servlet.LoginServlet.logIn(LoginServlet.java:275) at com.security.servlet.LoginServlet.doPost(LoginServlet.java:127) at com.security.servlet.LoginServlet.service(LoginServlet.java:432) at javax.servlet.http.HttpServlet.service(HttpServlet.java:803) at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:487) at org.mortbay.jetty.servlet.jsr77.Jsr77ServletHolder.handle(Jsr77ServletHolder.java:74) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:362) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:712) at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:405) at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:211) at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:139) at org.mortbay.jetty.Server.handle(Server.java:313) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:506) at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:844) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:644) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:205) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:381) at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:227) at org.mortbay.jetty.security.SslSocketConnector$SslConnection.run(SslSocketConnector.java:626) at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:442) Caused by: org.jboss.util.NestedSQLException: Could not enlist in transaction on entering meta-aware object!; - nested throwable: (javax.transaction.SystemException: java.lang.Throwable: Unabled to enlist resource, see the previous warnings. tx=TransactionImple < ac, BasicAction: -3f5702ff:38cc:47449b39:2e status: ActionStatus.ABORT_ONLY >); - nested throwable: (org.jboss.resource.JBossResourceException: Could not enlist in transaction on entering meta-aware object!; - nested throwable: (javax.transaction.SystemException: java.lang.Throwable: Unabled to enlist resource, see the previous warnings. tx=TransactionImple < ac, BasicAction: -3f5702ff:38cc:47449b39:2e status: ActionStatus.ABORT_ONLY >)) at org.jboss.resource.adapter.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:94) at com.common.ejb.DBConnection.getConnection(DBConnection.java:164) ... 68 more Caused by: org.jboss.resource.JBossResourceException: Could not enlist in transaction on entering meta-aware object!; - nested throwable: (javax.transaction.SystemException: java.lang.Throwable: Unabled to enlist resource, see the previous warnings. tx=TransactionImple < ac, BasicAction: -3f5702ff:38cc:47449b39:2e status: ActionStatus.ABORT_ONLY >) at org.jboss.resource.connectionmanager.TxConnectionManager.managedConnectionReconnected(TxConnectionManager.java:343) at org.jboss.resource.connectionmanager.BaseConnectionManager2.reconnectManagedConnection(BaseConnectionManager2.java:518) at org.jboss.resource.connectionmanager.BaseConnectionManager2.allocateConnection(BaseConnectionManager2.java:399) at org.jboss.resource.connectionmanager.BaseConnectionManager2$ConnectionManagerProxy.allocateConnection(BaseConnectionManager2.java:842) at org.jboss.resource.adapter.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:88) ... 69 more Caused by: javax.transaction.SystemException: java.lang.Throwable: Unabled to enlist resource, see the previous warnings. tx=TransactionImple < ac, BasicAction: -3f5702ff:38cc:47449b39:2e status: ActionStatus.ABORT_ONLY > at org.jboss.resource.connectionmanager.TxConnectionManager$TxConnectionEventListener$TransactionSynchronization.checkEnlisted(TxConnectionManager.java:744) at org.jboss.resource.connectionmanager.TxConnectionManager$TxConnectionEventListener.enlist(TxConnectionManager.java:577) at org.jboss.resource.connectionmanager.TxConnectionManager.managedConnectionReconnected(TxConnectionManager.java:337) ... 73 more 14:56:39,160 WARN [com.security.ejb.UserEJB.ejbLoad(UserEJB.java:257)] common.error.db.connectionfailed E The line failing in DBConnection is 'datasource.getConnection'. I will be attaching the appropriately sanitized log files shortly. There is currently no viable workaround. I will be posting the code used to create the Connection Pools and DataSources dynamically. If helpful, I can post the the 3. & 3.2 version as well as the 4.2 version. Regards, Steve