The CustomRunner used by ORM applies a mutual exclusive lookup when inspecting annotations like the @RequiresDialectFeature annotation. This meant that if a dialect feature requirement on the method successfully passed but the class had a requirement that would have failed; the test was actually executed anyway, which lead to some inconsistent behavior with tests after changes were made to the test suite to support HHH-13104 Resolved .

In short, the CustomRunner check should not apply a mutual exclusive lookup but rather gather the annotations from both the method and the class and check that all found annotations pass the check and in the event at least one fails; the method should be considered Ignored.