You can configure the JaasSecurityDomain in the AppServer\server\all\conf\jboss-service.xml file (create it if it doesn't exist). These pages might help:
http://docs.jboss.org/jbosssecurity/docs/6.0/security_guide/html/Login_Modules.html
http://docs.redhat.com/docs/en-US/JBoss_Enterprise_Application_Platform/5/html/Security_Guide/Using_LdapExtLoginModule_with_JaasSecurityDomain.html
One thing these examples fail to mention however is the <depends> attribute. Leaving it out caused the "JaasSecurityManager is not registered" error for me. Here's a working example:
<server>
<mbean code="org.jboss.security.plugins.JaasSecurityDomain" name="jboss.security:service=JaasSecurityDomain,domain=myDomain">
<constructor>
<arg type="java.lang.String" value="myDomain" />
</constructor>
<attribute name="KeyStoreURL">resource:test.jks</attribute>
<attribute name="KeyStorePass">password</attribute>
<depends>jboss.security:service=JaasSecurityManager</depends>
</mbean>
</server>