The general documentation can be found here: https://community.jboss.org/wiki/JBossAS7EnablingJASPIAuthenticationForWebApplications
But the current version of JBoss AS (7.1.1) contains various bugs, so it might still not actually work. I've written about this here: http://arjan-tijms.blogspot.nl/2012/11/implementing-container-authentication.html
See this for the one of the issues regarding the bugs: https://issues.jboss.org/browse/JBPAPP-10225
If I'm not mistaken there was another NPE bug in AS 7.1.1 that was fixed in 7.1.2 (which was not released as such, but can be downloaded as JBoss EAP 6.0.0 or build from source).