[hibernate-dev] Unit test failures using the new metamodel

[Gail Badner]

----- Original Message -----
> From: "Steve Ebersole" <steve at hibernate.org>
> To: "Gail Badner" <gbadner at redhat.com>
> Cc: "Hardy Ferentschik" <hardy at hibernate.org>, "Hibernate hibernate-dev" <hibernate-dev at lists.jboss.org>
> Sent: Monday, August 6, 2012 7:02:54 AM
> Subject: Re: [hibernate-dev] Unit test failures using the new metamodel
> 
> 
> On 08/02/2012 07:39 PM, Gail Badner wrote:
> >
> > IIRC, there were some tests that were annotated with
> > @FailureExpected that were only expected to fail using some
> > dialects. This was causing what looked like failures on the
> > dialects where the test was expected to pass. When we were trying
> > to clean up the test failures for the tested dialects, either
> > hibernate.test.validatefailureexpected was introduced or the
> > default for hibernate.test.validatefailureexpected was changed to
> > false. Steve might remember exactly why this was done.
> >
> > In any case, this is hindering more than helping when we're trying
> > to get things working with the new metamodel.
> 
> Like I said in my initial response that flag has always been there.
> The
> reason had more to do with different audiences running the tests IIRC
> .
> But like I said, it was years ago, so the exact specifics escape me.
> I'll search the dev list when i get a few minutes and see what I can
> dig up.
> 
> I think you are perhaps getting caught up in making changes for a
> transitory set of conditions.  We currently have a slew of failures
> mostly due to the tests using code they were never designed to test.

I'm not sure I understand what you mean? Do you mean that the FailureExpected code is not intended to expect failures in @Before, @BeforeClass, and @BeforeClassOnce callbacks?

> And even worse, that code is known to be incomplete.  The way
> "failure
> expected" is supposed to work is that tests which are marked as
> @FailureExpected are supposed to be handled the same as if they were
> marked with @Ignore (standard JUnit annotation) *if*
> `hibernate.test.validatefailureexpected` is not set.  If
> `hibernate.test.validatefailureexpected` is set that changes the
> logic
> up a bit.  However, again the problem is that we are leveraging tests
> to
> test something they were never designed to test.  In fact, they are
> so
> much not testing it that the code (now) under test occurs before the
> test really even begins.  There is supposed to be some code that
> skips
> before/after callbacks if all tests methods are being ignored.  It
> would
> seem that code is no longer working (or
> @FailureExpectedWithNewMetamodel
> was not hooked into that code completely) if the tests are marked for
> expected failure but before callbacks still happen.

Yes, this should be fixed.

> 
> Am I against changing `hibernate.test.validatefailureexpected` to
> default to true *for this branch, for the time being*?  No.  I
> already
> said I would be willing to discuss that.  But I'd also prefer to
> figure
> out why what is supposed to be happening in the hibernate-testing
> code
> is no longer happening.

+1

> 
> > When hibernate.test.validatefailureexpected is false (the default),
> > the result from FailureExpected tests is simply ignored.
> >> :-(
> Not understanding how this is not desirable?

If the result is simply ignored, then we don't know if changes would result in a @FailureExpectedWithNewMetamodel test to actually pass. 

With the validation flag set to true, a failure due to FailureExpectedTestPassedException would (hopefully)  mean that the @FailureExpectedWithNewMetamodel annotation can be removed. Once it's removed, we have a way to see if a future change breaks it.

> 
> >
> > Is there a way to shift the failure so it happens during test
> > execution so it would be handled properly?
> 
> Sure.  You would not use @Before, @BeforeClass, @BeforeClassOnce
> callbacks.  Instead you would build the metamodel during the test
> method
> execution.  But again, see above about tests testing code they were
> not
> designed to test...
>

I agree that this is not the way to fix this.

Thanks,
Gail
 
> 
> --
> steve at hibernate.org
> http://hibernate.org
> 
>