The crash recovery tests are automated in the AS testsuite as an optional task, see testsuite/imports/config/tests-crash-recovery.xml and JBAS-7219. They are included in both, the EAP5 branch and trunk.
Currently, the recovery tests run against selected databases inside the redhat network, namely PostgreSQL 8.2 and 8.3, Oracle 10g and 11g, Sybase 15, DB2 9.7, MSSQL 2005 and 2008, MySQL 5.0 and 5.1, i.e. supported db's for EAP. They don't use Derby at the moment.
The current tests involve only XA-datasource for each supported db in crash recovery and there is a question what other types of JCA XA resources they could involve into the recovery testing. It is subject for discussion.