I've got a similar problem using JBoss SOA 5.0.1.GA (which uses JBossTS 4.6.1.GA_CP03).
If I let the transaction manager move a transaction to the 'prepare' stage and then stop SQL Server, restarting it results in a 'suspect' database. I am then forced to abort the transaction manually before restarting the database server again. I then (sometimes) get the error that you see: -
16:08:03,777 WARN [loggerI18N] [com.arjuna.ats.internal.jta.recovery.xarecovery1] Local XARecoveryModule.xaRecovery got XA exception javax.transaction.xa.XAException: The functio
n RECOVER: has failed. The status is: -3. Error: "*** SQLJDBC_XA DTC_ERROR Context: xa_recover, state=1, StatusCode:-3 (0xFFFFFFFD) ***", XAException.XAER_RMERR
I haven't been able to repeatably reproduce this error though - sometimes the recovery manager successfully 'recovers' the transaction that I manually killed and you see this in the log: -
11:41:17,452 DEBUG [logger] Local XARecoveryModule.transactionInitiatedRecovery completed
11:41:17,483 DEBUG [AppServerJDBCXARecovery] Created new XAConnection
11:41:17,483 DEBUG [logger] xarecovery of XAResourceID:45
11:41:17,499 DEBUG [logger] Found 0 xids in doubt
This to me suggests that the recovery manager successfully sent the 'recover' command to SQL Server, requesting any in doubt transactions and that the server responded with none (because I had already aborted the transaction).