OK, so the reason we now throw InactiveTransactionException is because (as the JIRA states) IllegalStateException has a well defined meaning according to JTA "Thrown if the current thread is not associated with a transaction." In this case there is a transaction associated with the calling thread, it's just that it's no longer active. OTS correctly differentiates these cases, but once again JTA is broken in this regard ;)
So what we did was provide a new SystemException to try to allow for the differentiation in the case someone codes against our specific exceptions as well as those who don't care and only catch SystemExceptions (or not). I didn't like the idea of deriving from IllegalStateException because in the case of users who only catch that exception rather than, say, an InactiveTransactionException that is derived from ISE, their code may assume one thing (no transaction) when in fact that's not the case. At least with an exception derived from SE code cannot make any assumption about the thread-to-transaction association.