|
A test such as PerfTest is good to identify algorithmic issues or significant design mistakes like an N^3 could be on managing the number of entities in a session. By making it multithreaded it also becomes a good way to verify correct parallel execution (isolation) and allows to identify problematic locks but in that case you have to also make sure that each thread is able to validate the results it expects; ideally such a test should exercise all features in parallel so that you know that no combination is violating any assumption on parallelism.
But neither is a good way to state anything about expected end-user performance from it; if that is your goal, you need a driver (ideally external so that it can be run on independent hardware) which can inject a fixed load, for example simulating 600 users, where each of them will use the different capabilities in a reasonably weighted mixture. With such a test the performance you want to achieve is a parameter that you specify yourself (a parameter to the client) and what you measure is:
-
does the system behave correctly at that load (% of errors below some acceptable range, possibly zero)
-
is the response time for each request below the acceptability threshold
Ultimately that's the only kind of test which matters, if the goal is to validate OGM as production ready.. unfortunately that's a lot of work: being a library the flexibility is high and the combination of all different parallel operations can't be emulated by a close loop test such as PerfTest. If your goal is to validate algorithmic correctness of queries, I'd make an independent test for it which doesn't include other operations like CRUD in it as that would make it much harder to interpret the results.
|