Replying to my own reply ;-)
It's clear that the brain-dead JTA specification text around Transaction.setRollbackOnly doesn't consider the case of an inactive transaction when in fact it should, since to have a Transaction reference it doesn't make sense for IllegalStateException to mean "Thrown if the current thread is not associated with a transaction." If that were the case then Transaction should be null.
Plus it doesn't work in a multi-threaded environment, particularly if checked transaction semantics are disabled.
So it may make sense for derive from ISE rather than SE.
Let me sleep on this, or at least have a few beers.