[JBoss JIRA] Created: (JBTM-764) ConnectionImple/RecoverableConnection leak in ConnectionManager because of the pooling
by Mauro Molinari (JIRA)
ConnectionImple/RecoverableConnection leak in ConnectionManager because of the pooling
--------------------------------------------------------------------------------------
Key: JBTM-764
URL: https://jira.jboss.org/browse/JBTM-764
Project: JBoss Transaction Manager
Issue Type: Bug
Security Level: Public (Everyone can see)
Components: JTA
Affects Versions: 4.6.1
Environment: JBossTS JTA embedded in a webapp with Spring
Reporter: Mauro Molinari
Priority: Critical
This is indirectly related to JBTM-529.
Suppose you are using JBossTS JTA within your webapp, configured using Spring and using the DynamicClass mechanism to get connections.
Suppose you're doing connection pooling under JBossTS: that is, the DynamicClass implementation is pooling XA connections from the actual XADataSource provided by the DBMS JDBC driver. This is a different approach from the one described in JBTM-529 and we switched to this one because of the problems encountered with JBTM-529, thinking that it could be a more correct approach.
In this scenario, the client application code is doing the following:
- start a transaction
- request a new JDBC connection: this can be done either by using the DriverManager or by using Spring SimpleDriverDataSource; JBossTS JTA TransactionalDriver actually creates a new ConnectionImple, which asks the DynamicClass implementation to get a new XAConnection and gets a physical JDBC connection from that XAConnection; because of the XAConnection pooling provided by the DynamicClass implementation, this XAConnection might be a new one or a reused one, but I think this is not relevant to the problem
- do what you have to do with the connection
- close the connection: this causes the invocation of ConnectionImple.close
- do other things within the transaction
- commit the transaction
Now, what I see is this: com.arjuna.ats.internal.jdbc.ConnectionImple.close() does the following check:
if (!_recoveryConnection.inuse())
{
ConnectionManager.remove(this); // finalize?
}
The problem is that since the transaction is still active, _recoveryConnection.inuse() returns true, because it checks for the XAResource associated with it: in the case of DirectRecoverableConnection, this is done when com.arjuna.ats.internal.jdbc.DirectRecoverableConnection.reset() is invoked, that is when the RecoverableConnection is closed, and so when the transaction terminates.
The ultimate effect is that the ConnectionImple is never removed from the caching map of ConnectionManager, because it should have been when the connection was closed, BUT at that time it was not because its RecoverableConnection was in use.
What we're actually seeing in our application is that com.arjuna.ats.internal.jdbc.ConnectionManager._connections grows indefinitely. This causes a memory leak and severe performance problems, also because the test for cache hits in ConnectionManager is made using a linear iteration over the map (this could be improved a lot).
The workaround is to disable connection pooling within the transaction by using the patch from JBTM-529 (but also please see my last comment, too!).
To fix the problem I think that some approaches could be followed:
- the easiest one is to remove the old connections without any associated transaction when iterating over _connections: that is, if I'm checking a connection conn which has tx1 == null, I could remove conn from _connections; in this way I'm sure that sooner or later a ConnectionImple, which hasn't been removed from that map previously, will be when another connection is requested
- implement com.arjuna.ats.internal.jdbc.ConnectionImple.equals(Object) and hashCode() so that the call to _connections.put(conn, conn) will "replace any old (closed) connection which had the same connection information", as the inline documentation says; this is not true right now, because equality is based on identity; this might be the less invasive solution, but I don't know how easy it would be to do and if there may be regressions in other areas
- provide a mechanism to invoke com.arjuna.ats.internal.jdbc.ConnectionManager.remove(ConnectionImple) for all the ConnectionImple objects bound to a transaction when this is completed (committed or rolled back)
In any case, a smarter use of _connections would be useful for performance (using a Transaction=>ConnectionImple mapping, for instance...).
I would also consider the complete removal of in-transaction connection pooling: after all, every production use of JBossTS JTA will certainly use some sort of connection pooling by its own, so the benefits of using another level of pooling could be negligible. Even worse, performance could be compromised if the search for the hit in the pool is not efficient.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 10 months
[JBoss JIRA] Created: (JBTM-860) use XAResourceWrapper metadata for assume complete
by Jonathan Halliday (JIRA)
use XAResourceWrapper metadata for assume complete
--------------------------------------------------
Key: JBTM-860
URL: https://issues.jboss.org/browse/JBTM-860
Project: JBoss Transaction Manager
Issue Type: Feature Request
Security Level: Public (Everyone can see)
Components: Recovery
Affects Versions: 4.15.0
Reporter: Jonathan Halliday
Assignee: Jonathan Halliday
In the XA protocol a time window exists wherein the RM has committed and thus forgotten a tx branch but the TM has not yet deleted its log. A crash during this window currently results in an unrecoverable situation, as the TM assumes the branch belongs to an RM that is uncontactable and will retry recovery indefinitely. This stems from an inability to relate the Xid to a specific RM. It can be overridden globally with JTAEnvironmentBean.xaAssumeRecoveryComplete, but we would prefer more fine-grained control. With the availability of RM id information from XAResourceWrapper this becomes feasible.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 10 months
[JBoss JIRA] Created: (JBTM-789) Potential connection leak
by Mauro Molinari (JIRA)
Potential connection leak
-------------------------
Key: JBTM-789
URL: https://jira.jboss.org/browse/JBTM-789
Project: JBoss Transaction Manager
Issue Type: Bug
Security Level: Public (Everyone can see)
Components: JTA
Environment: JBoss Transactions 4.11, Tomcat, Spring
Reporter: Mauro Molinari
I recently upgraded JBossTS from 4.6.1 GA to 4.11 Final. I see that bug JBTM-532 that I opened against 4.5 and 4.6 should have been fixed in 4.8.0. However, I have a doubt on the current (4.11) implementation of com.arjuna.ats.internal.jdbc.ConnectionImple.close().
I mean, if _theModifier is null, a comment says: "no indication about connections, so assume close immediately". However in this case only _theConnection is closed and set to null, while _recoveryConnection.closeCloseCurrentConnection() is not called. I think this may lead to connection leaks.
What I would expect is that if _theModifier is null, a log entry were added but nothing else were done; in particular I would expect not to return and let the following "if (!delayClose)" (towards the end of the method) close the connections immediately (both _theConnection and _recoveryConnection).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 10 months
[JBoss JIRA] Created: (JBTM-529) Provide a way to disable connection pooling within a transaction
by Mauro Molinari (JIRA)
Provide a way to disable connection pooling within a transaction
----------------------------------------------------------------
Key: JBTM-529
URL: https://jira.jboss.org/jira/browse/JBTM-529
Project: JBoss Transaction Manager
Issue Type: Feature Request
Security Level: Public (Everyone can see)
Components: Configuration, Documentation, JTA
Affects Versions: 4.6.0, 4.5.0
Environment: JBossJTA + DataSource with connection pooling (i.e.: DBCP) + Spring
Reporter: Mauro Molinari
Priority: Critical
Attachments: patch.txt
Suppose you are using JBossTS within you webapp, configured using Spring.
Suppose you're doing connection pooling over JBossTS: that is, you have your data source with connection pooling managing connections obtained from the TransactionalDriver of JBossJTA.
As of now, com.arjuna.ats.jdbc.TransactionalDriver.connect(String, Properties) is calling com.arjuna.ats.internal.jdbc.ConnectionManager.create(String, Properties) to get a connection. The latter is behaving like this:
- if an "equivalent" connection has been already created, it's not closed and it is associated with the same transaction, return that connection
- otherwise create a new one
In other words, as stated by the comments in ConnectionManager source, this class is pooling connections for the duration of a transaction.
This however this can lead to severe and hard-to-find problems when you're doing your own connection pooling above JBossJTA using a data source like DBCP. These are two scenarios in which problems can occur:
1) suppose you have two different data sources that are modeling two logically different databases. However, suppose you're configuring both of them to point to the same physical database (that is: same URL, same username, same password) and you're off course using the same dynamic class to drive the TransactionalDriver to the actual JDBC driver.
Suppose transaction 1 (tx1) is started and, within its duration, a connection is asked to data source 1 (ds1) and another connection is asked to data source 2 (ds2). ds1 and ds2 are initially empty, so the request leads to the creation of two new connections, through TransactionalDriver.connect. Therefore, ds1 asks the TransactionalDriver for a new connection (conn1) and the TransactionalDriver asks the ConnectionManager to create it. conn1 is created and put into ds1, then made available to the client code. But also ds2 is requesting a new connection to the TransactionalDriver: this asks the ConnectionManager, which in turns realizes that it has been requested a connection for the same transaction (tx1), for the same URL, username and password and using the same dynamic class... then, instead of creating another connection (conn2), thanks to its pooling algorythm it returns again conn1, which then goes into ds2! Anyway, tx1 terminates without problems, because the physical database is actually the same.
However, after tx1 has terminated and new transactions are created, because conn1 is in both ds1 and ds2, it may happen that ds1 gives away conn1 to be used for a transaction tx2 while ds2 gives away conn1 to be used for another concurrent transaction tx3. Then, during tx2 or tx3 execution the "Checking transaction and found that this connection is already associated with a different transaction!" error occurs.
To work around this problem you could configure each data source to use a different implementation (that is, different class names!) of the dynamic class. However, another problem could still happen.
2) suppose you work around problem 1) by using a different dynamic class implementation for each data source. Now, suppose tx4 is started and, during its execution, a thread requests two different connections to ds1. Because of the pooling done by ConnectionManager, it may happen that ds1 receives conn1 twice from the TransactionalDriver, while thinking to receive two DIFFERENT connections. Now, suppose that the data source implementation can handle that without corrupting its internal data structures (and, based on my experiences with DBCP, I wouldn't bet on this!), however this causes conn1 to appear twice in ds1. Then, if tx5 and tx6 are two new concurrent transactions during which requests to ds1 are made, it can happen that conn1 is given to be used in both tx5 and tx6, producing again the already mentioned error.
So, my request is to support a configuration like the one I described by adding the ability to disable connection pooling at transaction level completely.
I'm attaching a suggested patch that provides this functionality by:
- adding support for a connection property named POOL_CONNECTIONS_WITHIN_TRANSACTION wich is assumed to be "true" by default but that can be set to "false" as needed
- changing ConnectionManager.create(String, Properties) to honour that property: if it is set to "false", pooling is disabled and a new connection is always created
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 10 months
[JBoss JIRA] Created: (JBTM-649) Support XAException.XA_RETRY in 2 phase commit -- Backward Compatibility issue
by Howard Gao (JIRA)
Support XAException.XA_RETRY in 2 phase commit -- Backward Compatibility issue
------------------------------------------------------------------------------
Key: JBTM-649
URL: https://jira.jboss.org/jira/browse/JBTM-649
Project: JBoss Transaction Manager
Issue Type: Feature Request
Security Level: Public (Everyone can see)
Affects Versions: 4.6.1
Reporter: Howard Gao
Fix For: 4.6.1.CP04
After upgrading from 4.4.0 to 4.6.1 (release for AS5), one of the JBoss Messaging test starts to fail. The scenario is:
1. We have two XAResources res1 and res2. Both are enlisted in a Transaction.
2. res1 sends a message to Queue1 (server1), res2 sends a message to Queue2 (server2).
3. delist res1 and res2 and commit the transaction.
4. During the commit process, we simulate a failure on res2.commit() so that the res1.prepare(), res2.prepare() and res1.commit() are all successful but res2.commit() will throw XAException.XA_RETRY exception -- eventually leaving the transaction in a Heuristic state.
5. Then we restore res2 back to normal, hoping that the Recovery Manager will kick in and recover the transaction. So we expect that the two sent messages will be received by the receiver. To allow the recovery to happen we give a 60 second timeout to receive the second message. But the result is the first message is received but the second message isn't.
We know that it is not JTA compliant. But some users may still rely on the old behavior to work.
Suggestion: a system property can be provided to enable the old behavior if specified by user.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 12 months
[JBoss JIRA] Created: (JBTM-622) Support for crash recovery of tx-inflow with multiple resource managers
by Toshiya Kobayashi (JIRA)
Support for crash recovery of tx-inflow with multiple resource managers
-----------------------------------------------------------------------
Key: JBTM-622
URL: https://jira.jboss.org/jira/browse/JBTM-622
Project: JBoss Transaction Manager
Issue Type: Feature Request
Security Level: Public (Everyone can see)
Components: JCA, Recovery, Transaction Core
Affects Versions: 4.7.0
Reporter: Toshiya Kobayashi
JCA tx-inflow spec allows the external EIS to provide the Xid for the inflowed transaction. The TM handling the inflowed (i.e. subordinate) transaction uses this Xid verbatim when talking to RMs.
In current implementation, each RM enlisted in the subordinate tx will see the same Xid as originally provided by the EIS. This screws recovery where multiple resources are involved, since a given Xid no longer belongs to exactly one RM, but may be shared by several. We can no longer return a single XAResource to drive the recovery for a given Xid.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 12 months
[JBoss JIRA] Created: (JBTM-849) registerSynchronization fails with from status message
by Carlo de Wolf (JIRA)
registerSynchronization fails with from status message
------------------------------------------------------
Key: JBTM-849
URL: https://issues.jboss.org/browse/JBTM-849
Project: JBoss Transaction Manager
Issue Type: Bug
Security Level: Public (Everyone can see)
Affects Versions: 4.15.1
Environment: https://github.com/jbossejb3/jboss-as/tree/991e836c1bcb6b471010c5930fc3f5...
Reporter: Carlo de Wolf
Attachments: JBTM-849-stracktrace.txt
registerSynchronization is called from within a beforeCompletion of another Synchronization. I would expect the status to be PREPARING (or anything but RUNNING).
{noformat}
java.lang.IllegalStateException: Synchronizations are not allowed! Transaction status isActionStatus.RUNNING
at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.registerSynchronizationImple(TransactionImple.java:367)
at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.registerSynchronization(TransactionImple.java:344)
at org.jboss.jca.core.connectionmanager.transaction.TransactionSynchronizer.getRegisteredSynchronizer(TransactionSynchronizer.java:198)
at org.jboss.jca.core.connectionmanager.listener.TxConnectionListener.enlist(TxConnectionListener.java:210)
at org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.managedConnectionReconnected(TxConnectionManagerImpl.java:407)
... 104 more
{noformat}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 12 months