On Fri, Apr 28, 2017 at 5:18 PM, Heiko W.Rupp <hrupp@redhat.com> wrote:

Test reliability (the above is part of this). We need to try to have more unit and also integration tests and make them more reliable. During the merge we saw test failures on developer machines while Travis was good. It turned out that this was due to timing. In the (RHQ) past we saw test failures because of test ordering. We should perhaps try to make our (integration) tests in random order on purpose, as in reality, the user will not run the code in the order we assume in tests either (yes, that may make setup and tear-down more complex).

+1 for not ordering tests, ie. making them stateless. It's also easier to understand from the reader pov.

IMO we should really focus on unit tests (I'm also writing that for myself, I admit I haven't written many). Currently our projects have much more integration tests than unit tests, which for my experience is surprising (I'm more used to having a ratio of 1:2 or 2:3 in favor of unit tests). While it can appear safer to rely more on integration tests, it has its downsides, we all know them: less predictive so more subject to "environmental failures" (corollary: we trust them less). They take longer to run. They're more difficult to debug. Sometimes they cannot be run in IDE (I like to run tests in IDE, it's more flexible than 'mvn test' and makes debugging easier).

For instance, when a query is tested with several possible flags, I think it would be nice to test only few of the possible cases through integration tests, and have a complete covering on unit tests where the logic lies. It will be much faster to run, provides almost the same covering and faster failure detection in case of something being broken with some of the flags.