It looks to me like hornetq is returning the wrong error code?
So you'd be better off moving your discussion to that forum.
XAER_RMERR is for a resource error inside the transaction, meaning
the transaction is in unknown state and can't be committed.
I think it should be using XAER_RMFAIL if the problem is a failure to enlist?
But ask in the transactions forum what the correct code should be,
since they are the ones that interpret them.