[hibernate-dev] Unit test failures using the new metamodel

[Steve Ebersole]

On Tue 07 Aug 2012 01:01:37 AM CDT, Gail Badner wrote:

>>> IIRC, there were some tests that were annotated with
>>> @FailureExpected that were only expected to fail using some
>>> dialects. This was causing what looked like failures on the
>>> dialects where the test was expected to pass. When we were trying
>>> to clean up the test failures for the tested dialects, either
>>> hibernate.test.validatefailureexpected was introduced or the
>>> default for hibernate.test.validatefailureexpected was changed to
>>> false. Steve might remember exactly why this was done.
>>>
>>> In any case, this is hindering more than helping when we're trying
>>> to get things working with the new metamodel.
>>
>> Like I said in my initial response that flag has always been there.
>> The
>> reason had more to do with different audiences running the tests IIRC
>> .
>> But like I said, it was years ago, so the exact specifics escape me.
>> I'll search the dev list when i get a few minutes and see what I can
>> dig up.
>>
>> I think you are perhaps getting caught up in making changes for a
>> transitory set of conditions.  We currently have a slew of failures
>> mostly due to the tests using code they were never designed to test.
>
> I'm not sure I understand what you mean? Do you mean that the FailureExpected code is not intended to expect failures in @Before, @BeforeClass, and @BeforeClassOnce callbacks?

That is exactly what I am saying.  You have to step back and look at 
how tests are structured.  And this is not specific to Hibernate tests 
*in any way*.  Test generally have (1) a set-up phase (2) a test phase 
and (3) a tear-down phase.  set-up and tear-down are meant to deal with 
things called test *fixtures*.  Google shows all kinds of great hits on 
definitions for test fixtures so i wont go into details.  But the gist 
is that building a test fixture is not at all expected to fail; its not 
the "thing under test".  This is the exactly what we have discussed a 
few times now.  If you are trying to "test something" the only way to 
do that reasonably well is to do it in that second "test phase".  The 
existing tests simply were not written on the premise they are now 
being twisted to.

In thinking about this more, I think this situation makes some sense.  
I was confused about why JUnit was just "eating" the failures.  IIUC 
JUnit is seeing failures in the set up phase and is simply ignoring the 
test.   What John and I discussed when we worked out the 
@FailureExpectedWithNewMetamodel was to possibly "store" start up 
failures (exceptions) and use them for each grouped test method.  I 
think that is not unreasonable, even long term, so I will work on that 
change after 4.1.6.

Part of this goes to the other email you sent out last night; I will 
respond to that separately.

>
>> And even worse, that code is known to be incomplete.  The way
>> "failure
>> expected" is supposed to work is that tests which are marked as
>> @FailureExpected are supposed to be handled the same as if they were
>> marked with @Ignore (standard JUnit annotation) *if*
>> `hibernate.test.validatefailureexpected` is not set.  If
>> `hibernate.test.validatefailureexpected` is set that changes the
>> logic
>> up a bit.  However, again the problem is that we are leveraging tests
>> to
>> test something they were never designed to test.  In fact, they are
>> so
>> much not testing it that the code (now) under test occurs before the
>> test really even begins.  There is supposed to be some code that
>> skips
>> before/after callbacks if all tests methods are being ignored.  It
>> would
>> seem that code is no longer working (or
>> @FailureExpectedWithNewMetamodel
>> was not hooked into that code completely) if the tests are marked for
>> expected failure but before callbacks still happen.
>
> Yes, this should be fixed.
>
>>
>> Am I against changing `hibernate.test.validatefailureexpected` to
>> default to true *for this branch, for the time being*?  No.  I
>> already
>> said I would be willing to discuss that.  But I'd also prefer to
>> figure
>> out why what is supposed to be happening in the hibernate-testing
>> code
>> is no longer happening.
>
> +1
>
>>
>>> When hibernate.test.validatefailureexpected is false (the default),
>>> the result from FailureExpected tests is simply ignored.
>>>> :-(
>> Not understanding how this is not desirable?
>
> If the result is simply ignored, then we don't know if changes would result in a @FailureExpectedWithNewMetamodel test to actually pass.
>
> With the validation flag set to true, a failure due to FailureExpectedTestPassedException would (hopefully)  mean that the @FailureExpectedWithNewMetamodel annotation can be removed. Once it's removed, we have a way to see if a future change breaks it.
>
>>
>>>
>>> Is there a way to shift the failure so it happens during test
>>> execution so it would be handled properly?
>>
>> Sure.  You would not use @Before, @BeforeClass, @BeforeClassOnce
>> callbacks.  Instead you would build the metamodel during the test
>> method
>> execution.  But again, see above about tests testing code they were
>> not
>> designed to test...
>>
>
> I agree that this is not the way to fix this.
>
> Thanks,
> Gail
>
>>
>> --
>> steve at hibernate.org
>> http://hibernate.org
>>
>>

--
steve at hibernate.org
http://hibernate.org