Re: [hibernate-dev] Unit test failures using the new metamodel

From: "Steve Ebersole" <steve(a)hibernate.org&gt; To: "Gail Badner" <gbadner(a)redhat.com&gt; Cc: "Hardy Ferentschik" <hardy(a)hibernate.org&gt;, "Hibernate hibernate-dev" <hibernate-dev(a)lists.jboss.org&gt; Sent: Monday, August 6, 2012 7:02:54 AM Subject: Re: [hibernate-dev] Unit test failures using the new metamodel On 08/02/2012 07:39 PM, Gail Badner wrote: > > IIRC, there were some tests that were annotated with > @FailureExpected that were only expected to fail using some > dialects. This was causing what looked like failures on the > dialects where the test was expected to pass. When we were trying > to clean up the test failures for the tested dialects, either > hibernate.test.validatefailureexpected was introduced or the > default for hibernate.test.validatefailureexpected was changed to > false. Steve might remember exactly why this was done. > > In any case, this is hindering more than helping when we're trying > to get things working with the new metamodel. Like I said in my initial response that flag has always been there. The reason had more to do with different audiences running the tests IIRC . But like I said, it was years ago, so the exact specifics escape me. I'll search the dev list when i get a few minutes and see what I can dig up. I think you are perhaps getting caught up in making changes for a transitory set of conditions. We currently have a slew of failures mostly due to the tests using code they were never designed to test.

And even worse, that code is known to be incomplete. The way "failure expected" is supposed to work is that tests which are marked as @FailureExpected are supposed to be handled the same as if they were marked with @Ignore (standard JUnit annotation) *if* `hibernate.test.validatefailureexpected` is not set. If `hibernate.test.validatefailureexpected` is set that changes the logic up a bit. However, again the problem is that we are leveraging tests to test something they were never designed to test. In fact, they are so much not testing it that the code (now) under test occurs before the test really even begins. There is supposed to be some code that skips before/after callbacks if all tests methods are being ignored. It would seem that code is no longer working (or @FailureExpectedWithNewMetamodel was not hooked into that code completely) if the tests are marked for expected failure but before callbacks still happen.

Am I against changing `hibernate.test.validatefailureexpected` to default to true *for this branch, for the time being*? No. I already said I would be willing to discuss that. But I'd also prefer to figure out why what is supposed to be happening in the hibernate-testing code is no longer happening.

> When hibernate.test.validatefailureexpected is false (the default), > the result from FailureExpected tests is simply ignored. >> :-( Not understanding how this is not desirable?

> > Is there a way to shift the failure so it happens during test > execution so it would be handled properly? Sure. You would not use @Before, @BeforeClass, @BeforeClassOnce callbacks. Instead you would build the metamodel during the test method execution. But again, see above about tests testing code they were not designed to test...

-- steve(a)hibernate.org http://hibernate.org