[JBoss JIRA] (JBTM-2390) port the RTS test org.jboss.jbossts.star.test.PerformanceTest to a benchmark
by Michael Musgrove (JIRA)
[ https://issues.jboss.org/browse/JBTM-2390?page=com.atlassian.jira.plugin.... ]
Michael Musgrove commented on JBTM-2390:
----------------------------------------
OK using horneq with (linux) asynchronous IO enabled and some other tweaks brings it to something more comparable (6 fold difference which when we factor in the extra http requests means a difference of 2-3 times), thanks:
{code}
Benchmark Mode Samples Score Score error Units
o.j.n.r.EmptyTxnTest.testEmptyTxn thrpt 1 1494.553 NaN ops/s
o.j.n.r.NoTxnTest.testNoTxn thrpt 1 5041.441 NaN ops/s
o.j.n.r.TxnTest.testTxn thrpt 1 215.502 NaN ops/s
{code}
Note, I cannot try it out with the volatile store since RTS needs to list all uids for recovery purposes (and the volatile store doesn't support this op).
> port the RTS test org.jboss.jbossts.star.test.PerformanceTest to a benchmark
> ----------------------------------------------------------------------------
>
> Key: JBTM-2390
> URL: https://issues.jboss.org/browse/JBTM-2390
> Project: JBoss Transaction Manager
> Issue Type: Task
> Reporter: Michael Musgrove
> Assignee: Michael Musgrove
> Fix For: 5.later
>
>
> Currently this test is run on an ad hoc basis on the NCL cluster. I would like it moved into the benchmark suite for more regular testing.
> I have started work on this task but am finding threads deadlocking which has delayed the port.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
9 years, 1 month
[JBoss JIRA] (JBTM-2394) Transaction Reaper "abuses" in-use connection, leading to RPC protocol corruption
by Tom Jenkinson (JIRA)
[ https://issues.jboss.org/browse/JBTM-2394?page=com.atlassian.jira.plugin.... ]
Tom Jenkinson commented on JBTM-2394:
-------------------------------------
I think we should move the discussion to the forum, you may find this class useful to consider: https://github.com/jbosstm/narayana/blob/master/ArjunaCore/arjuna/classes... It will alert you if a transaction is being completed while multiple threads are active in it. You can match uids with https://github.com/jbosstm/narayana/blob/master/ArjunaJTA/jta/classes/com...
If you do want to discuss it further I would definitely appreciate if you can move it to the forum so that it may act as a reference if other people in the future have similar requirements.
Thanks,
Tom
> Transaction Reaper "abuses" in-use connection, leading to RPC protocol corruption
> ---------------------------------------------------------------------------------
>
> Key: JBTM-2394
> URL: https://issues.jboss.org/browse/JBTM-2394
> Project: JBoss Transaction Manager
> Issue Type: Bug
> Reporter: Christian von Kutzleben
> Assignee: Tom Jenkinson
>
> The scenario is as follows:
> An regular JBoss application thread uses the connection associated with an XAResource for it's work with the database backend. Start(xid) had been invoked, which means, the associated connection is "switched" to that xid. After finishing work, the regular JBoss application thread would eventually call end(xid, TMSUCCESS).
> The application executes a query that takes longer than the configured timeout.
> The work with the database is done via database specific RPC invocations, caused by a enterprise bean (which uses the JPA API and the JPA implementation eventually talks to the database via the database protocol).
> The (lower-level) connection is a TCP/IP connection, and
> the RPC protocol if of the form: "send data to server, then wait for server reply"
> At this point it should be clear, that the TCP/IP connection should not be shared by another thread whilst in the middle of an RPC invocation, because neither does the database server expect any data at this moment on this particular TCP/IP connection nor does it work, that then 2 threads wait on the same socket to receive a reply. (E.g. a ClosedByInterruptException is likely, there might be other error though, depending how wrong data is interpreted, e.g. BufferUnderflowException).
> Unfortunately, this is exactly the behavior of the JBoss "Transaction Reaper" thread, that uses the very same XAResource, that had been used by the regular JBoss application thread and is currently associated to a connection, "switched" to the current xid.
> By definition of the XA specification, any other XAResource could be used to terminate that transaction branch, and it would be fine, if the transaction reaper thread would use any XAResource (regular one, or one from our recovery module) to do that.
> This is a conceptual flaw of the transaction reaper implementation, and we can't implement a workaround (except extremely silly things like reaper thread recognition by comparing thread names ...)
> We could synchronize access to the TCP/IP connection, to not allow another thread communication, while another thread is active doing so, however, that kind of defeats the purpose of the reaper thread, as it would be blocked indefinitely.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
9 years, 1 month
[JBoss JIRA] (JBTM-2394) Transaction Reaper "abuses" in-use connection, leading to RPC protocol corruption
by Christian von Kutzleben (JIRA)
[ https://issues.jboss.org/browse/JBTM-2394?page=com.atlassian.jira.plugin.... ]
Christian von Kutzleben commented on JBTM-2394:
-----------------------------------------------
"Alternatively, it would be possible for an XAResource implementation to detect concurrent access and use multiple (internal) connections to the database if that is a requirement of the implementation. "
I'm afraid, that a separate thread to connection mapping would make our XAResource implementation pretty unmaintainable.
Two threads attached to the same Xid is a very special corner case (btw. the reaper does detect and report this in the log),
it would be easy for us to implement a separate physical connection, if the reaper could identify itself as a "special" thread.
"A caller invokes TX::enlist(XAResource). Callers do not provide us with "connections" or factories (I quote the word to signify it draw out how these can be JDBC/JMS/etc) to attempt to obtain an XAR from."
Yes, that is indeed a problem of the JTA. I totally agree with you, that it is a technical problem to acquire an XAResource for such purposes,
IMO JTA should be updated to provide such functionality, maybe a simple clone() specification for XAResource would be sufficient.
(Btw. I updated my previous comment to include XAConnection references, while you were writing your reply)
> Transaction Reaper "abuses" in-use connection, leading to RPC protocol corruption
> ---------------------------------------------------------------------------------
>
> Key: JBTM-2394
> URL: https://issues.jboss.org/browse/JBTM-2394
> Project: JBoss Transaction Manager
> Issue Type: Bug
> Reporter: Christian von Kutzleben
> Assignee: Tom Jenkinson
>
> The scenario is as follows:
> An regular JBoss application thread uses the connection associated with an XAResource for it's work with the database backend. Start(xid) had been invoked, which means, the associated connection is "switched" to that xid. After finishing work, the regular JBoss application thread would eventually call end(xid, TMSUCCESS).
> The application executes a query that takes longer than the configured timeout.
> The work with the database is done via database specific RPC invocations, caused by a enterprise bean (which uses the JPA API and the JPA implementation eventually talks to the database via the database protocol).
> The (lower-level) connection is a TCP/IP connection, and
> the RPC protocol if of the form: "send data to server, then wait for server reply"
> At this point it should be clear, that the TCP/IP connection should not be shared by another thread whilst in the middle of an RPC invocation, because neither does the database server expect any data at this moment on this particular TCP/IP connection nor does it work, that then 2 threads wait on the same socket to receive a reply. (E.g. a ClosedByInterruptException is likely, there might be other error though, depending how wrong data is interpreted, e.g. BufferUnderflowException).
> Unfortunately, this is exactly the behavior of the JBoss "Transaction Reaper" thread, that uses the very same XAResource, that had been used by the regular JBoss application thread and is currently associated to a connection, "switched" to the current xid.
> By definition of the XA specification, any other XAResource could be used to terminate that transaction branch, and it would be fine, if the transaction reaper thread would use any XAResource (regular one, or one from our recovery module) to do that.
> This is a conceptual flaw of the transaction reaper implementation, and we can't implement a workaround (except extremely silly things like reaper thread recognition by comparing thread names ...)
> We could synchronize access to the TCP/IP connection, to not allow another thread communication, while another thread is active doing so, however, that kind of defeats the purpose of the reaper thread, as it would be blocked indefinitely.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
9 years, 1 month
[JBoss JIRA] (JBTM-2390) port the RTS test org.jboss.jbossts.star.test.PerformanceTest to a benchmark
by Tom Jenkinson (JIRA)
[ https://issues.jboss.org/browse/JBTM-2390?page=com.atlassian.jira.plugin.... ]
Tom Jenkinson commented on JBTM-2390:
-------------------------------------
I am surprised you think its an anomaly. As an example, when you turn volatile store on our performance is something much more than a 15 fold improvement iirc?
Some suggestions:
1. Are you using HQ store (can you use it in this config)?
2. Have you tried enabling the volatile store?
In terms of notx vs emptytx I expect that is all down to the creation of the TX endpoints. You could profile the server to get more details there. I imagine your participants are no-ops so in the no tx case you have 2 no-op calls vs the empty tx case where you have 3 no-ops (I will class commit() here as a no-op as I assume it really is very little) and 1 op (the creation of the tx endpoints). Seems reasonable to me.
In the emptytx case, can you drop the commit calls (without it falling over) to see the cost of the commit vs the create?
> port the RTS test org.jboss.jbossts.star.test.PerformanceTest to a benchmark
> ----------------------------------------------------------------------------
>
> Key: JBTM-2390
> URL: https://issues.jboss.org/browse/JBTM-2390
> Project: JBoss Transaction Manager
> Issue Type: Task
> Reporter: Michael Musgrove
> Assignee: Michael Musgrove
> Fix For: 5.later
>
>
> Currently this test is run on an ad hoc basis on the NCL cluster. I would like it moved into the benchmark suite for more regular testing.
> I have started work on this task but am finding threads deadlocking which has delayed the port.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
9 years, 1 month
[JBoss JIRA] (JBTM-2394) Transaction Reaper "abuses" in-use connection, leading to RPC protocol corruption
by Christian von Kutzleben (JIRA)
[ https://issues.jboss.org/browse/JBTM-2394?page=com.atlassian.jira.plugin.... ]
Christian von Kutzleben edited comment on JBTM-2394 at 5/8/15 9:45 AM:
-----------------------------------------------------------------------
Hi Tom,
My interpretation of JTA 1.0.1, section 3.4.3 is a bit different from yours:
IMO it focuses on "Different Java threads may be using the same connection resource to access the
resource manager if the connection spans multiple method invocation." and not on "threads, XAResources, anything goes!"
It does not specifically advocate that the current transaction reaper implementation is a viable approach.
We can synchronize access to the connection currently associated with this XAResource, however then the
transaction reaper thread can't do it's job. This change would be spec-compliant, the RPC protocol would
not get corrupted, still, the reaper would not work.
Edit: please also see the javadoc for XAConnection: "An XAConnection object may be enlisted in a distributed transaction by means of an XAResource object. A transaction manager, usually part of a middle tier server, manages an XAConnection object through the XAResource object." and the javadoc for PooledConnection's (base of XAConnection) getConnection() method.
With this and a strict interpretation of JTA 3.4.3 one could imply, that access to the result of getConnection() (or it's underlying physical channel) should be synchronized; and if it is implemented that way, the reaper would be blocked forever.
was (Author: cvk):
Hi Tom,
My interpretation of JTA 1.0.1, section 3.4.3 is a bit different from yours:
IMO it focuses on "Different Java threads may be using the same connection resource to access the
resource manager if the connection spans multiple method invocation." and not on "threads, XAResources, anything goes!"
It does not specifically advocate that the current transaction reaper implementation is a viable approach.
We can synchronize access to the connection currently associated with this XAResource, however then the
transaction reaper thread can't do it's job. This change would be spec-compliant, the RPC protocol would
not get corrupted, still, the reaper would not work.
> Transaction Reaper "abuses" in-use connection, leading to RPC protocol corruption
> ---------------------------------------------------------------------------------
>
> Key: JBTM-2394
> URL: https://issues.jboss.org/browse/JBTM-2394
> Project: JBoss Transaction Manager
> Issue Type: Bug
> Reporter: Christian von Kutzleben
> Assignee: Tom Jenkinson
>
> The scenario is as follows:
> An regular JBoss application thread uses the connection associated with an XAResource for it's work with the database backend. Start(xid) had been invoked, which means, the associated connection is "switched" to that xid. After finishing work, the regular JBoss application thread would eventually call end(xid, TMSUCCESS).
> The application executes a query that takes longer than the configured timeout.
> The work with the database is done via database specific RPC invocations, caused by a enterprise bean (which uses the JPA API and the JPA implementation eventually talks to the database via the database protocol).
> The (lower-level) connection is a TCP/IP connection, and
> the RPC protocol if of the form: "send data to server, then wait for server reply"
> At this point it should be clear, that the TCP/IP connection should not be shared by another thread whilst in the middle of an RPC invocation, because neither does the database server expect any data at this moment on this particular TCP/IP connection nor does it work, that then 2 threads wait on the same socket to receive a reply. (E.g. a ClosedByInterruptException is likely, there might be other error though, depending how wrong data is interpreted, e.g. BufferUnderflowException).
> Unfortunately, this is exactly the behavior of the JBoss "Transaction Reaper" thread, that uses the very same XAResource, that had been used by the regular JBoss application thread and is currently associated to a connection, "switched" to the current xid.
> By definition of the XA specification, any other XAResource could be used to terminate that transaction branch, and it would be fine, if the transaction reaper thread would use any XAResource (regular one, or one from our recovery module) to do that.
> This is a conceptual flaw of the transaction reaper implementation, and we can't implement a workaround (except extremely silly things like reaper thread recognition by comparing thread names ...)
> We could synchronize access to the TCP/IP connection, to not allow another thread communication, while another thread is active doing so, however, that kind of defeats the purpose of the reaper thread, as it would be blocked indefinitely.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
9 years, 1 month
[JBoss JIRA] (JBTM-2394) Transaction Reaper "abuses" in-use connection, leading to RPC protocol corruption
by Tom Jenkinson (JIRA)
[ https://issues.jboss.org/browse/JBTM-2394?page=com.atlassian.jira.plugin.... ]
Tom Jenkinson commented on JBTM-2394:
-------------------------------------
In that scenario the reaper would attempt to rollback the XAR a number of times (detecting the wedged resource) and finally it would give up trying to call rollback on the XAR and just call setRollbackOnly on the TX. I can say that the transaction reaper has been working in this manner successfully with many resource managers over the years.
The reason we added JBTM-2342 is to assist with JPA which specifically is not thread-safe and the app server integration is required to call EntityManager.close() during afterCompletion(). You mentioned that you cannot use reaper thread names - this is entirely true and was considered during discussions on our forum regarding JBTM-2342. It was determined that this would not work in all scenarios as it is possible that a timed out transaction in a remote parent coordinator _may_ fire before the reaper anyway and in that case the rollback would come via (for example) Jacorb RequestProcessors and would be undetectable as a concurrent call. Hence JBTM-2342 provides an SPI to assist further.
Alternatively, it would be possible for an XAResource implementation to detect concurrent access and use multiple (internal) connections to the database if that is a requirement of the implementation.
I honestly feel that you will get better advice by discussing this on the forum. The transaction manager component (so Narayana) is not intended to be aware of connections. A caller invokes TX::enlist(XAResource). Callers do not provide us with "connections" or factories (I quote the word to signify it draw out how these can be JDBC/JMS/etc) to attempt to obtain an XAR from.
Finally, as you point out the recovery manager is able to obtain XAR from helpers but the recovery manager has a very specific purpose and is not intended for this use case. Not that I would advocate a change in this area but if you were considering such a thing, determining which of the XAR registered with the recovery manager should be used to issue the rollback would be impossible as it stands.
> Transaction Reaper "abuses" in-use connection, leading to RPC protocol corruption
> ---------------------------------------------------------------------------------
>
> Key: JBTM-2394
> URL: https://issues.jboss.org/browse/JBTM-2394
> Project: JBoss Transaction Manager
> Issue Type: Bug
> Reporter: Christian von Kutzleben
> Assignee: Tom Jenkinson
>
> The scenario is as follows:
> An regular JBoss application thread uses the connection associated with an XAResource for it's work with the database backend. Start(xid) had been invoked, which means, the associated connection is "switched" to that xid. After finishing work, the regular JBoss application thread would eventually call end(xid, TMSUCCESS).
> The application executes a query that takes longer than the configured timeout.
> The work with the database is done via database specific RPC invocations, caused by a enterprise bean (which uses the JPA API and the JPA implementation eventually talks to the database via the database protocol).
> The (lower-level) connection is a TCP/IP connection, and
> the RPC protocol if of the form: "send data to server, then wait for server reply"
> At this point it should be clear, that the TCP/IP connection should not be shared by another thread whilst in the middle of an RPC invocation, because neither does the database server expect any data at this moment on this particular TCP/IP connection nor does it work, that then 2 threads wait on the same socket to receive a reply. (E.g. a ClosedByInterruptException is likely, there might be other error though, depending how wrong data is interpreted, e.g. BufferUnderflowException).
> Unfortunately, this is exactly the behavior of the JBoss "Transaction Reaper" thread, that uses the very same XAResource, that had been used by the regular JBoss application thread and is currently associated to a connection, "switched" to the current xid.
> By definition of the XA specification, any other XAResource could be used to terminate that transaction branch, and it would be fine, if the transaction reaper thread would use any XAResource (regular one, or one from our recovery module) to do that.
> This is a conceptual flaw of the transaction reaper implementation, and we can't implement a workaround (except extremely silly things like reaper thread recognition by comparing thread names ...)
> We could synchronize access to the TCP/IP connection, to not allow another thread communication, while another thread is active doing so, however, that kind of defeats the purpose of the reaper thread, as it would be blocked indefinitely.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
9 years, 1 month
[JBoss JIRA] (JBTM-2390) port the RTS test org.jboss.jbossts.star.test.PerformanceTest to a benchmark
by Michael Musgrove (JIRA)
[ https://issues.jboss.org/browse/JBTM-2390?page=com.atlassian.jira.plugin.... ]
Michael Musgrove commented on JBTM-2390:
----------------------------------------
The port is giving me unusual results which I am investigating (I'll update the JIRA with the outcome). I am finding that a transaction with two participants is running 15 times more slowly than one with no participants. The number of requests in the two scenarios differ in the ratio 10:4 and one of them is doing disk writes but I would not expect that to account for a 15 fold difference. The two test scenarios are as follows:
# client starts a (RTS) transaction and sends requests to two non transactional services.
# client starts a (RTS) transaction and sends requests to two transactional services.
The first scenario requires 4 http requests:
* 1 request to the coordinator to begin a transaction
* 2 non transactional service requests
* 1 request to the coordinator to end the transaction
The second scenario requires 10 http requests:
* 1 request to the coordinator to begin a transaction
* 2 transactional service requests
** 2 participant enlist requests to the coordinator
* 1 request to the coordinator to end the transaction
** 2 prepare requests (one for each enlisted participant)
** 2 commit requests
The benchmark output (on my laptop) for these two scenarios (I include the case of no transaction for completeness) is:
{code}
java -jar ArjunaJTA/jta/target/benchmarks.jar "org.jboss.narayana.rts.*TxnTest.*" -i 1 -wi 0 -f 1 -t 1 -r 100
Benchmark Mode Samples Score Score error Units
o.j.n.r.EmptyTxnTest.testEmptyTxn thrpt 1 1447.812 NaN ops/s
o.j.n.r.NoTxnTest.testNoTxn thrpt 1 4720.317 NaN ops/s
o.j.n.r.TxnTest.testTxn thrpt 1 90.956 NaN ops/s
{code}
The benchmark with no transaction (two service requests) versus a transaction with no participants is slightly puzzling too since neither involves disk writing, uses half as many http requests and yet is 3 times faster (should have been twice as fast).
I am running the coordinator and services on Undertow. I did try configs where the services were collocated with the coordinator and then when they were on different Undertow servers with no significant change in behaviour.
> port the RTS test org.jboss.jbossts.star.test.PerformanceTest to a benchmark
> ----------------------------------------------------------------------------
>
> Key: JBTM-2390
> URL: https://issues.jboss.org/browse/JBTM-2390
> Project: JBoss Transaction Manager
> Issue Type: Task
> Reporter: Michael Musgrove
> Assignee: Michael Musgrove
> Fix For: 5.later
>
>
> Currently this test is run on an ad hoc basis on the NCL cluster. I would like it moved into the benchmark suite for more regular testing.
> I have started work on this task but am finding threads deadlocking which has delayed the port.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
9 years, 1 month
[JBoss JIRA] (JBTM-2394) Transaction Reaper "abuses" in-use connection, leading to RPC protocol corruption
by Christian von Kutzleben (JIRA)
[ https://issues.jboss.org/browse/JBTM-2394?page=com.atlassian.jira.plugin.... ]
Christian von Kutzleben commented on JBTM-2394:
-----------------------------------------------
Hi Tom,
My interpretation of JTA 1.0.1, section 3.4.3 is a bit different from yours:
IMO it focuses on "Different Java threads may be using the same connection resource to access the
resource manager if the connection spans multiple method invocation." and not on "threads, XAResources, anything goes!"
It does not specifically advocate that the current transaction reaper implementation is a viable approach.
We can synchronize access to the connection currently associated with this XAResource, however then the
transaction reaper thread can't do it's job. This change would be spec-compliant, the RPC protocol would
not get corrupted, still, the reaper would not work.
> Transaction Reaper "abuses" in-use connection, leading to RPC protocol corruption
> ---------------------------------------------------------------------------------
>
> Key: JBTM-2394
> URL: https://issues.jboss.org/browse/JBTM-2394
> Project: JBoss Transaction Manager
> Issue Type: Bug
> Reporter: Christian von Kutzleben
> Assignee: Tom Jenkinson
>
> The scenario is as follows:
> An regular JBoss application thread uses the connection associated with an XAResource for it's work with the database backend. Start(xid) had been invoked, which means, the associated connection is "switched" to that xid. After finishing work, the regular JBoss application thread would eventually call end(xid, TMSUCCESS).
> The application executes a query that takes longer than the configured timeout.
> The work with the database is done via database specific RPC invocations, caused by a enterprise bean (which uses the JPA API and the JPA implementation eventually talks to the database via the database protocol).
> The (lower-level) connection is a TCP/IP connection, and
> the RPC protocol if of the form: "send data to server, then wait for server reply"
> At this point it should be clear, that the TCP/IP connection should not be shared by another thread whilst in the middle of an RPC invocation, because neither does the database server expect any data at this moment on this particular TCP/IP connection nor does it work, that then 2 threads wait on the same socket to receive a reply. (E.g. a ClosedByInterruptException is likely, there might be other error though, depending how wrong data is interpreted, e.g. BufferUnderflowException).
> Unfortunately, this is exactly the behavior of the JBoss "Transaction Reaper" thread, that uses the very same XAResource, that had been used by the regular JBoss application thread and is currently associated to a connection, "switched" to the current xid.
> By definition of the XA specification, any other XAResource could be used to terminate that transaction branch, and it would be fine, if the transaction reaper thread would use any XAResource (regular one, or one from our recovery module) to do that.
> This is a conceptual flaw of the transaction reaper implementation, and we can't implement a workaround (except extremely silly things like reaper thread recognition by comparing thread names ...)
> We could synchronize access to the TCP/IP connection, to not allow another thread communication, while another thread is active doing so, however, that kind of defeats the purpose of the reaper thread, as it would be blocked indefinitely.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
9 years, 1 month