Hi Judes.
I am struggling with the same issue trying to migrate an application from JBoss 4.2.2 to 5.1.0.
The application uses EJB based web services without any extra descriptors (web.xml / jboss-web.xml) and worked like a charm with authentication in 4.2.2.
In 5.1.0 authentication always fails. The @SecurityDomain annotations just seems to be ignored and some kind of fallback to the "other" security domain is made.
I guess it has something to do with this issue: https://issues.jboss.org/browse/JBAS-7037.