[hibernate-dev] Unit test failures using the new metamodel

[Steve Ebersole]

On 08/02/2012 07:39 PM, Gail Badner wrote:
>
> IIRC, there were some tests that were annotated with @FailureExpected that were only expected to fail using some dialects. This was causing what looked like failures on the dialects where the test was expected to pass. When we were trying to clean up the test failures for the tested dialects, either hibernate.test.validatefailureexpected was introduced or the default for hibernate.test.validatefailureexpected was changed to false. Steve might remember exactly why this was done.
>
> In any case, this is hindering more than helping when we're trying to get things working with the new metamodel.

Like I said in my initial response that flag has always been there. The 
reason had more to do with different audiences running the tests IIRC .  
But like I said, it was years ago, so the exact specifics escape me.  
I'll search the dev list when i get a few minutes and see what I can dig up.

I think you are perhaps getting caught up in making changes for a 
transitory set of conditions.  We currently have a slew of failures 
mostly due to the tests using code they were never designed to test.  
And even worse, that code is known to be incomplete.  The way "failure 
expected" is supposed to work is that tests which are marked as 
@FailureExpected are supposed to be handled the same as if they were 
marked with @Ignore (standard JUnit annotation) *if* 
`hibernate.test.validatefailureexpected` is not set.  If 
`hibernate.test.validatefailureexpected` is set that changes the logic 
up a bit.  However, again the problem is that we are leveraging tests to 
test something they were never designed to test.  In fact, they are so 
much not testing it that the code (now) under test occurs before the 
test really even begins.  There is supposed to be some code that skips 
before/after callbacks if all tests methods are being ignored.  It would 
seem that code is no longer working (or @FailureExpectedWithNewMetamodel 
was not hooked into that code completely) if the tests are marked for 
expected failure but before callbacks still happen.

Am I against changing `hibernate.test.validatefailureexpected` to 
default to true *for this branch, for the time being*?  No.  I already 
said I would be willing to discuss that.  But I'd also prefer to figure 
out why what is supposed to be happening in the hibernate-testing code 
is no longer happening.

> When hibernate.test.validatefailureexpected is false (the default),
> the result from FailureExpected tests is simply ignored.
>> :-(
Not understanding how this is not desirable?

>
> Is there a way to shift the failure so it happens during test execution so it would be handled properly?

Sure.  You would not use @Before, @BeforeClass, @BeforeClassOnce 
callbacks.  Instead you would build the metamodel during the test method 
execution.  But again, see above about tests testing code they were not 
designed to test...


-- 
steve at hibernate.org
http://hibernate.org