September 2013 - jbossts-issues - Jboss List Archives

[JBoss JIRA] (JBTM-1763) Consider running the QA tests in parrallel

by Tom Jenkinson (JIRA)

[ https://issues.jboss.org/browse/JBTM-1763?page=com.atlassian.jira.plugin.... ] Tom Jenkinson updated JBTM-1763: -------------------------------- Fix Version/s: 6.0.0.Final > Consider running the QA tests in parrallel > ------------------------------------------ > > Key: JBTM-1763 > URL: https://issues.jboss.org/browse/JBTM-1763 > Project: JBoss Transaction Manager > Issue Type: Feature Request > Security Level: Public(Everyone can see) > Components: Testing > Reporter: Paul Robinson > Assignee: Tom Jenkinson > Priority: Minor > Fix For: 6.0.0.Final > > > The QA tests currently take 5hrs to run per orb. This constitutes 83% of our CI run time. > My understanding is that the time is mostly taken up in Thread.sleep. The test times range from 1 to 99seconds. So there are no major hotspots. The problem is that there are so many tests. > I suspect we could run these tests in parallel to make better utilisation of resources. > The tests don't run in an AS, but they do acquire ports. There may be other restrictions that make running them in parallel hard or impossible. > It would be easier to simply carve up the tests into, say 5 parts per orb and have CI do the parallelism. However, this will consume more CI nodes than if we ran them in parallel on a single node. > I think it would be worth investigating if we can run the tests in n batches (say n=10 initially, but should be configurable). Where each batch runs with a different port configuration. This may prove too disruptive to the tests, > and also more bother than it is worth. Maybe we could do an initial investigation to see if this has legs? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira

11 years, 2 months

1
0
0 / 0

[JBoss JIRA] (JBTM-1801) Move all test properties into root pom.xml

by Tom Jenkinson (JIRA)

[ https://issues.jboss.org/browse/JBTM-1801?page=com.atlassian.jira.plugin.... ] Tom Jenkinson updated JBTM-1801: -------------------------------- Fix Version/s: 6.0.0.Final (was: 5.0.0.M5) > Move all test properties into root pom.xml > ------------------------------------------ > > Key: JBTM-1801 > URL: https://issues.jboss.org/browse/JBTM-1801 > Project: JBoss Transaction Manager > Issue Type: Feature Request > Security Level: Public(Everyone can see) > Components: Build System > Reporter: Paul Robinson > Assignee: Gytis Trikleris > Priority: Trivial > Fix For: 6.0.0.Final > > > We have these test properties duplicated in many places: > https://github.com/jbosstm/narayana/blob/master/XTS/localjunit/pom.xml#L22 > We should move them up to the root pom.xml to simplify maintenance. As part of this task you need to hunt out all these usages and similar. Then come up with a common set of properties that works in all cases and move to the root pom.xml. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira

11 years, 2 months

1
0
0 / 0

[JBoss JIRA] (JBTM-1763) Consider running the QA tests in parrallel

by Tom Jenkinson (JIRA)

[ https://issues.jboss.org/browse/JBTM-1763?page=com.atlassian.jira.plugin.... ] Tom Jenkinson updated JBTM-1763: -------------------------------- Fix Version/s: (was: 5.0.0.M5) > Consider running the QA tests in parrallel > ------------------------------------------ > > Key: JBTM-1763 > URL: https://issues.jboss.org/browse/JBTM-1763 > Project: JBoss Transaction Manager > Issue Type: Feature Request > Security Level: Public(Everyone can see) > Components: Testing > Reporter: Paul Robinson > Assignee: Tom Jenkinson > Priority: Minor > > The QA tests currently take 5hrs to run per orb. This constitutes 83% of our CI run time. > My understanding is that the time is mostly taken up in Thread.sleep. The test times range from 1 to 99seconds. So there are no major hotspots. The problem is that there are so many tests. > I suspect we could run these tests in parallel to make better utilisation of resources. > The tests don't run in an AS, but they do acquire ports. There may be other restrictions that make running them in parallel hard or impossible. > It would be easier to simply carve up the tests into, say 5 parts per orb and have CI do the parallelism. However, this will consume more CI nodes than if we ran them in parallel on a single node. > I think it would be worth investigating if we can run the tests in n batches (say n=10 initially, but should be configurable). Where each batch runs with a different port configuration. This may prove too disruptive to the tests, > and also more bother than it is worth. Maybe we could do an initial investigation to see if this has legs? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira

11 years, 2 months

1
0
0 / 0

[JBoss JIRA] (JBTM-1454) Compare performance of REST-AT with JTS

by Tom Jenkinson (JIRA)

[ https://issues.jboss.org/browse/JBTM-1454?page=com.atlassian.jira.plugin.... ] Tom Jenkinson updated JBTM-1454: -------------------------------- Fix Version/s: 6.0.0.Final (was: 5.0.0.Final) > Compare performance of REST-AT with JTS > --------------------------------------- > > Key: JBTM-1454 > URL: https://issues.jboss.org/browse/JBTM-1454 > Project: JBoss Transaction Manager > Issue Type: Task > Security Level: Public(Everyone can see) > Components: JTS, Performance Testing, REST > Reporter: Tom Jenkinson > Assignee: Michael Musgrove > Priority: Minor > Fix For: 6.0.0.Final > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira

11 years, 2 months

1
0
0 / 0

[JBoss JIRA] (JBTM-764) ConnectionImple/RecoverableConnection leak in ConnectionManager because of the pooling

by Tom Jenkinson (JIRA)

[ https://issues.jboss.org/browse/JBTM-764?page=com.atlassian.jira.plugin.s... ] Tom Jenkinson updated JBTM-764: ------------------------------- Fix Version/s: 6.0.0.Final (was: 5.0.0.Final) > ConnectionImple/RecoverableConnection leak in ConnectionManager because of the pooling > -------------------------------------------------------------------------------------- > > Key: JBTM-764 > URL: https://issues.jboss.org/browse/JBTM-764 > Project: JBoss Transaction Manager > Issue Type: Bug > Security Level: Public(Everyone can see) > Components: Resource Manager > Affects Versions: 4.6.1, 4.11.0 > Environment: JBossTS JTA embedded in a webapp with Spring > Reporter: Mauro Molinari > Assignee: Tom Jenkinson > Priority: Critical > Fix For: 6.0.0.Final > > > This is indirectly related to JBTM-529. > Suppose you are using JBossTS JTA within your webapp, configured using Spring and using the DynamicClass mechanism to get connections. > Suppose you're doing connection pooling under JBossTS: that is, the DynamicClass implementation is pooling XA connections from the actual XADataSource provided by the DBMS JDBC driver. This is a different approach from the one described in JBTM-529 and we switched to this one because of the problems encountered with JBTM-529, thinking that it could be a more correct approach. > In this scenario, the client application code is doing the following: > - start a transaction > - request a new JDBC connection: this can be done either by using the DriverManager or by using Spring SimpleDriverDataSource; JBossTS JTA TransactionalDriver actually creates a new ConnectionImple, which asks the DynamicClass implementation to get a new XAConnection and gets a physical JDBC connection from that XAConnection; because of the XAConnection pooling provided by the DynamicClass implementation, this XAConnection might be a new one or a reused one, but I think this is not relevant to the problem > - do what you have to do with the connection > - close the connection: this causes the invocation of ConnectionImple.close > - do other things within the transaction > - commit the transaction > Now, what I see is this: com.arjuna.ats.internal.jdbc.ConnectionImple.close() does the following check: > if (!_recoveryConnection.inuse()) > { > ConnectionManager.remove(this); // finalize? > } > The problem is that since the transaction is still active, _recoveryConnection.inuse() returns true, because it checks for the XAResource associated with it: in the case of DirectRecoverableConnection, this is done when com.arjuna.ats.internal.jdbc.DirectRecoverableConnection.reset() is invoked, that is when the RecoverableConnection is closed, and so when the transaction terminates. > The ultimate effect is that the ConnectionImple is never removed from the caching map of ConnectionManager, because it should have been when the connection was closed, BUT at that time it was not because its RecoverableConnection was in use. > What we're actually seeing in our application is that com.arjuna.ats.internal.jdbc.ConnectionManager._connections grows indefinitely. This causes a memory leak and severe performance problems, also because the test for cache hits in ConnectionManager is made using a linear iteration over the map (this could be improved a lot). > The workaround is to disable connection pooling within the transaction by using the patch from JBTM-529 (but also please see my last comment, too!). > To fix the problem I think that some approaches could be followed: > - the easiest one is to remove the old connections without any associated transaction when iterating over _connections: that is, if I'm checking a connection conn which has tx1 == null, I could remove conn from _connections; in this way I'm sure that sooner or later a ConnectionImple, which hasn't been removed from that map previously, will be when another connection is requested > - implement com.arjuna.ats.internal.jdbc.ConnectionImple.equals(Object) and hashCode() so that the call to _connections.put(conn, conn) will "replace any old (closed) connection which had the same connection information", as the inline documentation says; this is not true right now, because equality is based on identity; this might be the less invasive solution, but I don't know how easy it would be to do and if there may be regressions in other areas > - provide a mechanism to invoke com.arjuna.ats.internal.jdbc.ConnectionManager.remove(ConnectionImple) for all the ConnectionImple objects bound to a transaction when this is completed (committed or rolled back) > In any case, a smarter use of _connections would be useful for performance (using a Transaction=>ConnectionImple mapping, for instance...). > I would also consider the complete removal of in-transaction connection pooling: after all, every production use of JBossTS JTA will certainly use some sort of connection pooling by its own, so the benefits of using another level of pooling could be negligible. Even worse, performance could be compromised if the search for the hit in the pool is not efficient. > PLEASE NOTE: this problem has been observed in our production environment, using JBossTS JTA 4.6.1_GA. I did some browsing of JBossTS JTA 4.11.0_Final and although some little changes (like the use of a HashSet instead of a Hashtable) it seems to me that the problem is still there. I couldn't do further investigation for now, but I will do soon. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira

11 years, 2 months

1
0
0 / 0

[JBoss JIRA] (JBTM-1227) StAX is not available on Android

by Tom Jenkinson (JIRA)

[ https://issues.jboss.org/browse/JBTM-1227?page=com.atlassian.jira.plugin.... ] Tom Jenkinson updated JBTM-1227: -------------------------------- Fix Version/s: 6.0.0.Final (was: 5.0.0.Final) > StAX is not available on Android > -------------------------------- > > Key: JBTM-1227 > URL: https://issues.jboss.org/browse/JBTM-1227 > Project: JBoss Transaction Manager > Issue Type: Task > Security Level: Public(Everyone can see) > Components: Common > Affects Versions: 4.16.4, 5.0.0.M1 > Environment: Android 2.3.3 > Reporter: Mark Little > Assignee: Gytis Trikleris > Fix For: 6.0.0.Final > > > StAX is not available. Since we use it for property configuration, the system fails to run unmodified. There is an equivalent implementation (XmlPullParser) but it is likely other JBoss projects will suffer from this issue. > See http://www.ibm.com/developerworks/library/x-android/ for more details. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira

11 years, 2 months

1
0
0 / 0

[JBoss JIRA] (JBTM-1454) Compare performance of REST-AT with JTS

by Tom Jenkinson (JIRA)

[ https://issues.jboss.org/browse/JBTM-1454?page=com.atlassian.jira.plugin.... ] Tom Jenkinson updated JBTM-1454: -------------------------------- Component/s: BlackTie > Compare performance of REST-AT with JTS > --------------------------------------- > > Key: JBTM-1454 > URL: https://issues.jboss.org/browse/JBTM-1454 > Project: JBoss Transaction Manager > Issue Type: Task > Security Level: Public(Everyone can see) > Components: BlackTie, JTS, Performance Testing, REST > Reporter: Tom Jenkinson > Assignee: Michael Musgrove > Priority: Minor > Fix For: 6.0.0.Final > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira

11 years, 2 months

1
0
0 / 0

[JBoss JIRA] (JBTM-789) XA connections leak when no modifier available

by Tom Jenkinson (JIRA)

[ https://issues.jboss.org/browse/JBTM-789?page=com.atlassian.jira.plugin.s... ] Tom Jenkinson updated JBTM-789: ------------------------------- Fix Version/s: 6.0.0.Final (was: 5.0.0.Final) > XA connections leak when no modifier available > ---------------------------------------------- > > Key: JBTM-789 > URL: https://issues.jboss.org/browse/JBTM-789 > Project: JBoss Transaction Manager > Issue Type: Bug > Security Level: Public(Everyone can see) > Components: Resource Manager > Affects Versions: 4.11.0 > Environment: JBoss Transactions 4.11, Tomcat, Spring > Reporter: Mauro Molinari > Assignee: Tom Jenkinson > Priority: Critical > Fix For: 6.0.0.Final > > Attachments: Patch JBTM-789.txt > > > I recently upgraded JBossTS from 4.6.1 GA to 4.11 Final. I see that bug JBTM-532 that I opened against 4.5 and 4.6 should have been fixed in 4.8.0. However, besides still having a null _theModifier when using PostgreSQL, the current (4.11) implementation of com.arjuna.ats.internal.jdbc.ConnectionImple.close() has a problem. > > I mean, if _theModifier is null, a comment says: "no indication about connections, so assume close immediately". However in this case only _theConnection (java.sql.Connection) is closed and set to null, while _recoveryConnection.closeCloseCurrentConnection() is not called, leading to an XAConnection leak. > What I would expect is that if _theModifier is null, a log entry were added but nothing else were done; in particular I would expect not to return and let the following "if (!delayClose)" (towards the end of the method) close both the connections immediately. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira

11 years, 2 months

1
0
0 / 0

[JBoss JIRA] (JBTM-1928) Add last x00 lines of build log in the comment when a pull fails

by Tom Jenkinson (JIRA)

[ https://issues.jboss.org/browse/JBTM-1928?page=com.atlassian.jira.plugin.... ] Tom Jenkinson commented on JBTM-1928: ------------------------------------- This would help a lot for external contributors. > Add last x00 lines of build log in the comment when a pull fails > ---------------------------------------------------------------- > > Key: JBTM-1928 > URL: https://issues.jboss.org/browse/JBTM-1928 > Project: JBoss Transaction Manager > Issue Type: Task > Security Level: Public(Everyone can see) > Components: Build System > Reporter: Tom Jenkinson > Assignee: Gytis Trikleris > Priority: Critical > Fix For: 5.0.0.M5 > > > It would be useful for external contributors if when the pull request fails, the comment included the last few hundred lines of output so that it might provide some context to the issue. As it stands an internal engineer has to check the log -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira

11 years, 2 months

1
0
0 / 0

[JBoss JIRA] (JBTM-531) com.arjuna.ats.internal.jdbc.DynamicClass.shutdownDataSource(XADataSource) is never called

by Tom Jenkinson (JIRA)

[ https://issues.jboss.org/browse/JBTM-531?page=com.atlassian.jira.plugin.s... ] Tom Jenkinson updated JBTM-531: ------------------------------- Fix Version/s: 6.0.0.Final (was: 5.0.0.Final) > com.arjuna.ats.internal.jdbc.DynamicClass.shutdownDataSource(XADataSource) is never called > ------------------------------------------------------------------------------------------ > > Key: JBTM-531 > URL: https://issues.jboss.org/browse/JBTM-531 > Project: JBoss Transaction Manager > Issue Type: Bug > Security Level: Public(Everyone can see) > Components: Resource Manager > Affects Versions: 4.5.0, 4.6.0 > Reporter: Mauro Molinari > Assignee: Tom Jenkinson > Priority: Critical > Fix For: 6.0.0.Final > > > com.arjuna.ats.internal.jdbc.DynamicClass.shutdownDataSource(XADataSource) is never called by com.arjuna.ats.internal.jdbc.DirectRecoverableConnection. > From what I could understand (but I'm not sure of this), after calling com.arjuna.ats.internal.jdbc.DirectRecoverableConnection.closeCloseCurrentConnection() the DirectRecoverableConnection won't be used anymore, so that method could be a good candidate to call _dynamicConnection.shutDownDataSource(_theDataSource) and to null both _dynamicConnection and _theDataSource. > This problem could be quite severe, because if the XADataSource needs to be shut down in order to release resources, this could lead to leaks. > Moreover, I was wondering if a caching of the dynamic class wouldn't be a good idea... as of now, a NEW XA data source is created every time a new connection is requested... and this could be expensive... -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira

11 years, 2 months

1
0
0 / 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

jbossts-issues September 2013