With the introduction of https://jira.jboss.org/browse/JBTM-234 we see regression on EJB 3 testsuites when setRollbackOnly is called from afterCompletion. We expect to get an IllegalStateException, instead a SystemException is thrown.
As the issue states OTS raises Inactive:
OTS 1.4 2.6.14 rollback_only
The transaction associated with the target object is modified so that the only possible
outcome is to rollback the transaction. The Inactive exception is raised if the
transaction has already been prepared.
While the JTA spec doesn't explicitly account for such a scenario.
To recount:
void beforeCompletion()
The beforeCompletion method is called by the transaction manager prior to the start of the two-phase transaction commit process. This call is executed with the transaction context of the transaction that is being committed.
void afterCompletion(int status)
This method is called by the transaction manager after the transaction is committed or rolled back.
And
The aftercompletion callback will be invoked in an undefined context.
void setRollbackOnly() throws java.lang.IllegalStateException, SystemException
IllegalStateException - Thrown if the target object is not associated with any transaction.
SystemException - Thrown if the transaction manager encounters an unexpected error condition.
Now it really becomes a choice which exception should be thrown.
public class SystemException extends java.lang.Exception
The SystemException is thrown by the transaction manager to indicate that it has encountered an unexpected error condition that prevents future transaction services from proceeding.
In principal both the state of the transaction and transaction services are not compromised and transaction services proceeds in a defined manner. The outcome of the transaction is the status as passed into the afterCompletion.
Based on the description in SystemException I would say that TS should not throw that exception, but instead the IllegalStateException.