[Hawkular-dev] Performance problems? (was RxJava2 preliminary testing)

[Filip Brychta]

----- Original Message -----
> Hi,
> 
> Testing revealed something else that worried me, I was quite sure that
> I've had far higher numbers before than what we have now, so I went back
> to test some older versions to see if this was true..
> 
> The testing setup is as follows: 4 cores, 8GB, ccm Cassandra (3.0.10,
> 3.10 made no significant difference here). So all the CPU freed from
> HWKMETRICS is used by Cassandra (they compete on resources). Uses
> core-metrics-service only, no REST interface - directly writing with
> RxJava using addDatapoints.
> 
> 1 or 10 datapoint per metric, writing 100 000 metrics in one call to
> addDatapoints (insertBenchmark, not the -Single ending one). Sources in
> the jmh-benchmark module (fix_jmh branch, PR #

REST equivalent to previous call would be a POST request containing 100 000 metrics with 1 or 10 datapoints for each metric? Is that a possible scenario?
Is it possible to compare perf results (not absolute numbers but at least trends) from tests bypassing the REST interface with end-to-end perf tests?
Your results show huge drop which is not visible in my results from end-to-end test. Since 0.19.3 I can see only two drops caused by PR652 and PR710. In total about 20% drop (for small msgs).

> 0.19.3:
> 
> 1 datapoint -> 31004 metrics / second (31004 datapoints per second)
> 
> 10 datapoints -> 19027 metrics / second (190270 datapoints per second)
> 
> Current master:
> 
> 1 datapoint -> 8535 metrics / second (8535 datapoints per second)
> 
> 10 datapoints -> 7065 metrics / second (70650 datapoints per second)
> 
> So performance has dropped significantly between current master and
> 0.19.3 (0.19.0 was the first release without double writing).
> 
> With HWKMETRICS-599 (micro-batching on top of the current master):
> 
> RxJava1:
> 
> 1 datapoint -> 55036 metrics / second (55036 datapoints / second)
> 
> 10 datapoints -> 17870 metrics / second (178700 datapoints / second)
> 
> RxJava2:
> 
> 1 datapoint -> 76216 metrics / second (76216 datapoints / second)
> 
> 10 datapoints -> 20088 metrics / second (200880 datapoints / second)
> 
> HWKMETRICS-599 without retryWhen made no difference (an assumption that
> this was the problem creator).
> 
> The 1 datapoint per metric is the most interesting use-case, so that's
> something we should concentrate on. But before someone asks:
> 
> 100 metrics, 1000 datapoints per metric and batch size of 1000 (instead
> of default 50 in the micro-batching):
> 
> 298030 datapoints / second.
> 
> That's all folks!
> 
>    - Micke
> 
> On 02/11/2017 12:04 PM, Michael Burman wrote:
> > Hi,
> >
> > I did yesterday evening and today some testing on how using RxJava2
> > would benefit us (I'm expecting more from RxJava 2.1 actually, since it
> > has some enhanced parallelism features which we might benefit from).
> >
> > Short notes from RxJava2 migration, it's more painful than I assumed.
> > The code changes can be small in terms of lines of code changed, but
> > almost every method has had their signature or behavior changed. So at
> > least I've had to read the documentation all the time when doing things
> > and trying to unlearn what I've done in the RxJava1.
> >
> > And all this comes with a backwards compatibility pressure for Java 6
> > (so you can't benefit from many Java 8 advantages). Reactive-Commons /
> > Reactor have started from Java 8 to provide cleaner implementation. Grr.
> >
> > I wrote a simple write path modification in the PR #762 (metrics) that
> > writes Gauges using RxJava2 ported micro-batching feature. There's still
> > some RxJavaInterOp use in it, so that might slow down the performance a
> > little bit. However, it is possible to merge these two codes. There are
> > also some other optimizations I think could be worth it.
> >
> > I'd advice against it though, reading gets quite complex. I would almost
> > suggest that we would do the MetricsServiceImpl/DataAccessImpl merging
> > by rewriting small parts at a time in the new class with RxJava2 and
> > make that call the old code with RxJavaInterOp. That way we could move
> > slowly to the newer codebase.
> >
> > I fixed the JMH-benchmarks (as they're not compiled in our CI and were
> > actually broken by some other PRs) and ran some tests. These are the
> > tests that measure only the metrics-core-service performance and do not
> > touch the REST-interface (or Wildfly) at all, thus giving better
> > comparison in how our internal changes behave.
> >
> > What I'm seeing is around 20-30% difference in performance when writing
> > gauges this way. So this should offset some of the issues we saw when we
> > improved error handling (which caused performance degradation). I did
> > ran into the HWKMETRICS-542 (BusyPoolException) so the tests were run
> > with 1024 connections.
> >
> > I'll continue next week some more testing, but at the same time I proved
> > that the micro-batching features do improve performance in the internal
> > processing, especially when there's small amount of writers to a single
> > node. But testing those features could probably benefit from more
> > benchmark tests without WIldfly (which takes so much processing power
> > that most performance improvements can't be measured correctly anymore).
> >
> >      - Micke
> > _______________________________________________
> > hawkular-dev mailing list
> > hawkular-dev at lists.jboss.org
> > https://lists.jboss.org/mailman/listinfo/hawkular-dev
> 
> _______________________________________________
> hawkular-dev mailing list
> hawkular-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/hawkular-dev
>