[jbossts-issues] [JBoss JIRA] (JBTM-2394) Transaction Reaper "abuses" in-use connection, leading to RPC protocol corruption

[Christian von Kutzleben (JIRA)]

     [ https://issues.jboss.org/browse/JBTM-2394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Christian von Kutzleben updated JBTM-2394:
------------------------------------------
    Description: 
The scenario is as follows:

An regular JBoss application thread uses the connection associated with an XAResource for it's work with the database backend. Start(xid) had been invoked, which means, the associated connection is "switched" to that xid. After finishing work, the regular JBoss application thread would eventually call end(xid, TMSUCCESS).

The application executes a query that takes longer than the configured timeout.

The work with the database is done via database specific RPC invocations, caused by a enterprise bean (which uses the JPA API and the JPA implementation eventually talks to the database via the database protocol).

The (lower-level) connection is --unsurprisingly-- a TCP/IP connection, and
the RPC protocol --also unsurprisingly-- if of the form: "send data to server, then wait for server reply"

At this point it should be clear, that the TCP/IP connection should not be shared by another thread whilst in the middle of an RPC invocation, because neither does the database server expect any data at this moment on this particular TCP/IP connection nor does it work, that then 2 threads wait on the same socket to receive a reply. (E.g. a ClosedByInterruptException is likely, there might be other error though, depending how wrong data is interpreted, e.g. BufferUnderflowException).

Unfortunately, this is exactly the behavior of the JBoss "Transaction Reaper" thread, that uses the very same XAResource, that had been used by the regular JBoss application thread and is currently associated to a connection, "switched" to the current xid.

By definition of the XA specification, any other XAResource could be used to terminate that transaction branch, and it would be fine, if the transaction reaper thread would use any XAResource (regular one, or one from our recovery module) to do that.

This is a conceptual flaw of the transaction reaper implementation, and we can't implement a workaround (except extremely silly things like reaper thread recognition by comparing thread names ...)

We could synchronized access to the TCP/IP connection, to not allow another thread communication, while another thread is active doing so, however, that kind of defeats the purpose of the reaper thread, as it would be blocked indefinitely.

  was:
The scenario is as follows:

An regular JBoss application thread uses the connection associated with an XAResource for it's work with the database backend. Start(xid) had been invoked, which means, the associated connection is "switched" to that xid. After finishing work, the regular JBoss application thread would eventually call end(xid, TMSUCCESS).

The work with the database is done via database specific RPC invocations, caused by a enterprise bean (which uses the JPA API and the JPA implementation eventually talks to the database via the database protocol).

The (lower-level) connection is --unsurprisingly-- a TCP/IP connection, and
the RPC protocol --also unsurprisingly-- if of the form: "send data to server, then wait for server reply"

At this point it should be clear, that the TCP/IP connection should not be shared by another thread whilst in the middle of an RPC invocation, because neither does the database server expect any data at this moment on this particular TCP/IP connection nor does it work, that then 2 threads wait on the same socket to receive a reply. (E.g. a ClosedByInterruptException is likely, there might be other error though, depending how wrong data is interpreted, e.g. BufferUnderflowException).

Unfortunately, this is exactly the behavior of the JBoss "Transaction Reaper" thread, that uses the very same XAResource, that had been used by the regular JBoss application thread and is currently associated to a connection, "switched" to the current xid.

By definition of the XA specification, any other XAResource could be used to terminate that transaction branch, and it would be fine, if the transaction reaper thread would use any XAResource (regular one, or one from our recovery module) to do that.

This is a conceptual flaw of the transaction reaper implementation, and we can't implement a workaround (except extremely silly things like reaper thread recognition by comparing thread names ...)

We could synchronized access to the TCP/IP connection, to not allow another thread communication, while another thread is active doing so, however, that kind of defeats the purpose of the reaper thread, as it would be blocked indefinitely.



> Transaction Reaper "abuses" in-use connection, leading to RPC protocol corruption
> ---------------------------------------------------------------------------------
>
>                 Key: JBTM-2394
>                 URL: https://issues.jboss.org/browse/JBTM-2394
>             Project: JBoss Transaction Manager
>          Issue Type: Bug
>            Reporter: Christian von Kutzleben
>            Assignee: Tom Jenkinson
>
> The scenario is as follows:
> An regular JBoss application thread uses the connection associated with an XAResource for it's work with the database backend. Start(xid) had been invoked, which means, the associated connection is "switched" to that xid. After finishing work, the regular JBoss application thread would eventually call end(xid, TMSUCCESS).
> The application executes a query that takes longer than the configured timeout.
> The work with the database is done via database specific RPC invocations, caused by a enterprise bean (which uses the JPA API and the JPA implementation eventually talks to the database via the database protocol).
> The (lower-level) connection is --unsurprisingly-- a TCP/IP connection, and
> the RPC protocol --also unsurprisingly-- if of the form: "send data to server, then wait for server reply"
> At this point it should be clear, that the TCP/IP connection should not be shared by another thread whilst in the middle of an RPC invocation, because neither does the database server expect any data at this moment on this particular TCP/IP connection nor does it work, that then 2 threads wait on the same socket to receive a reply. (E.g. a ClosedByInterruptException is likely, there might be other error though, depending how wrong data is interpreted, e.g. BufferUnderflowException).
> Unfortunately, this is exactly the behavior of the JBoss "Transaction Reaper" thread, that uses the very same XAResource, that had been used by the regular JBoss application thread and is currently associated to a connection, "switched" to the current xid.
> By definition of the XA specification, any other XAResource could be used to terminate that transaction branch, and it would be fine, if the transaction reaper thread would use any XAResource (regular one, or one from our recovery module) to do that.
> This is a conceptual flaw of the transaction reaper implementation, and we can't implement a workaround (except extremely silly things like reaper thread recognition by comparing thread names ...)
> We could synchronized access to the TCP/IP connection, to not allow another thread communication, while another thread is active doing so, however, that kind of defeats the purpose of the reaper thread, as it would be blocked indefinitely.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)