[Hawkular-dev] Some food for thought about improving the release of (large) features

[Joel Takvorian]

On Fri, Apr 28, 2017 at 5:18 PM, Heiko W.Rupp <hrupp at redhat.com> wrote:
>
>
>    -
>
>    Test reliability (the above is part of this). We need to try to have
>    more unit and also integration tests and make them more reliable. During
>    the merge we saw test failures on developer machines while Travis was good.
>    It turned out that this was due to timing. In the (RHQ) past we saw test
>    failures because of test ordering. We should perhaps try to make our
>    (integration) tests in random order on purpose, as in reality, the user
>    will not run the code in the order we assume in tests either (yes, that may
>    make setup and tear-down more complex).
>
> +1 for not ordering tests, ie. making them stateless. It's also easier to
understand from the reader pov.

IMO we should really focus on unit tests (I'm also writing that for myself,
I admit I haven't written many). Currently our projects have much more
integration tests than unit tests, which for my experience is surprising
(I'm more used to having a ratio of 1:2 or 2:3 in favor of unit tests).
While it can appear safer to rely more on integration tests, it has its
downsides, we all know them: less predictive so more subject to
"environmental failures" (corollary: we trust them less). They take longer
to run. They're more difficult to debug. Sometimes they cannot be run in
IDE (I like to run tests in IDE, it's more flexible than 'mvn test' and
makes debugging easier).

For instance, when a query is tested with several possible flags, I think
it would be nice to test only few of the possible cases through integration
tests, and have a complete covering on unit tests where the logic lies. It
will be much faster to run, provides almost the same covering and faster
failure detection in case of something being broken with some of the flags.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/hawkular-dev/attachments/20170502/0f7746f3/attachment.html