I think if we are going to include more performance test we should think about what we want to achieve and how we can avoid some of the problems we had with the current performance module.

First off, I don't see a point in having a performance and performance1 module. Really things should get rolled up into a single module. Here are a few other thoughts:

• The performance report should go into a dedicated file and not just to standard out. ATM, depending on how you run the tests the output is buried in different file. A dedicated performance result logger could be used for that. It should not matter whether I run the tests standalone or in container, the performance results should be in a fixed location.
• The used Search version should be configurable, eg via profiles. The Module could define multiple profiles for different Search versions (the default being the current SNAPSHOT version). This way one can easily run the harness against previous versions. Absolute numbers for a given version are imo not very helpful, because they are very hardware dependent. However, it makes sense to run against different Search versions and compare results between them. This was the harness can be used to spot regressions. If the output of the run is in a dedicated file, I can also just diff multiple runs (against various versions) to detect regressions
• The code should compile and run in a minimal configuration as part of the build. Since we are talking unit test it should be possible to just do a single loop in the minimalistic case. This ensures that the code compiles and the tests actually run. We had cases before where the tests would not run or even worth not compile, because the performance module was excluded from the main build