you should for sure not re-create the ValidatorFactory on each request.

Yes, we re-create the ValidatorFactory for the sake of the test. Originally, we cached not only the ValidatorFactory but also the Validator (as documentation states is thread safe), and never had a problem in JBoss. Problems started when we migrate to Weblogic.