Re: [Hawkular-dev] Performance problems? (was RxJava2 preliminary testing)

Hi, Testing revealed something else that worried me, I was quite sure that I've had far higher numbers before than what we have now, so I went back to test some older versions to see if this was true.. The testing setup is as follows: 4 cores, 8GB, ccm Cassandra (3.0.10, 3.10 made no significant difference here). So all the CPU freed from HWKMETRICS is used by Cassandra (they compete on resources). Uses core-metrics-service only, no REST interface - directly writing with RxJava using addDatapoints. 1 or 10 datapoint per metric, writing 100 000 metrics in one call to addDatapoints (insertBenchmark, not the -Single ending one). Sources in the jmh-benchmark module (fix_jmh branch, PR #

0.19.3: 1 datapoint -> 31004 metrics / second (31004 datapoints per second) 10 datapoints -> 19027 metrics / second (190270 datapoints per second) Current master: 1 datapoint -> 8535 metrics / second (8535 datapoints per second) 10 datapoints -> 7065 metrics / second (70650 datapoints per second) So performance has dropped significantly between current master and 0.19.3 (0.19.0 was the first release without double writing). With HWKMETRICS-599 (micro-batching on top of the current master): RxJava1: 1 datapoint -> 55036 metrics / second (55036 datapoints / second) 10 datapoints -> 17870 metrics / second (178700 datapoints / second) RxJava2: 1 datapoint -> 76216 metrics / second (76216 datapoints / second) 10 datapoints -> 20088 metrics / second (200880 datapoints / second) HWKMETRICS-599 without retryWhen made no difference (an assumption that this was the problem creator). The 1 datapoint per metric is the most interesting use-case, so that's something we should concentrate on. But before someone asks: 100 metrics, 1000 datapoints per metric and batch size of 1000 (instead of default 50 in the micro-batching): 298030 datapoints / second. That's all folks! - Micke On 02/11/2017 12:04 PM, Michael Burman wrote: > Hi, > > I did yesterday evening and today some testing on how using RxJava2 > would benefit us (I'm expecting more from RxJava 2.1 actually, since it > has some enhanced parallelism features which we might benefit from). > > Short notes from RxJava2 migration, it's more painful than I assumed. > The code changes can be small in terms of lines of code changed, but > almost every method has had their signature or behavior changed. So at > least I've had to read the documentation all the time when doing things > and trying to unlearn what I've done in the RxJava1. > > And all this comes with a backwards compatibility pressure for Java 6 > (so you can't benefit from many Java 8 advantages). Reactive-Commons / > Reactor have started from Java 8 to provide cleaner implementation. Grr. > > I wrote a simple write path modification in the PR #762 (metrics) that > writes Gauges using RxJava2 ported micro-batching feature. There's still > some RxJavaInterOp use in it, so that might slow down the performance a > little bit. However, it is possible to merge these two codes. There are > also some other optimizations I think could be worth it. > > I'd advice against it though, reading gets quite complex. I would almost > suggest that we would do the MetricsServiceImpl/DataAccessImpl merging > by rewriting small parts at a time in the new class with RxJava2 and > make that call the old code with RxJavaInterOp. That way we could move > slowly to the newer codebase. > > I fixed the JMH-benchmarks (as they're not compiled in our CI and were > actually broken by some other PRs) and ran some tests. These are the > tests that measure only the metrics-core-service performance and do not > touch the REST-interface (or Wildfly) at all, thus giving better > comparison in how our internal changes behave. > > What I'm seeing is around 20-30% difference in performance when writing > gauges this way. So this should offset some of the issues we saw when we > improved error handling (which caused performance degradation). I did > ran into the HWKMETRICS-542 (BusyPoolException) so the tests were run > with 1024 connections. > > I'll continue next week some more testing, but at the same time I proved > that the micro-batching features do improve performance in the internal > processing, especially when there's small amount of writers to a single > node. But testing those features could probably benefit from more > benchmark tests without WIldfly (which takes so much processing power > that most performance improvements can't be measured correctly anymore). > > - Micke > _______________________________________________ > hawkular-dev mailing list > hawkular-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/hawkular-dev _______________________________________________ hawkular-dev mailing list hawkular-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/hawkular-dev