[Hawkular-dev] Performance problems? (was RxJava2 preliminary testing)

[Michael Burman]

Hi,

Testing revealed something else that worried me, I was quite sure that 
I've had far higher numbers before than what we have now, so I went back 
to test some older versions to see if this was true..

The testing setup is as follows: 4 cores, 8GB, ccm Cassandra (3.0.10, 
3.10 made no significant difference here). So all the CPU freed from 
HWKMETRICS is used by Cassandra (they compete on resources). Uses 
core-metrics-service only, no REST interface - directly writing with 
RxJava using addDatapoints.

1 or 10 datapoint per metric, writing 100 000 metrics in one call to 
addDatapoints (insertBenchmark, not the -Single ending one). Sources in 
the jmh-benchmark module (fix_jmh branch, PR #

0.19.3:

1 datapoint -> 31004 metrics / second (31004 datapoints per second)

10 datapoints -> 19027 metrics / second (190270 datapoints per second)

Current master:

1 datapoint -> 8535 metrics / second (8535 datapoints per second)

10 datapoints -> 7065 metrics / second (70650 datapoints per second)

So performance has dropped significantly between current master and 
0.19.3 (0.19.0 was the first release without double writing).

With HWKMETRICS-599 (micro-batching on top of the current master):

RxJava1:

1 datapoint -> 55036 metrics / second (55036 datapoints / second)

10 datapoints -> 17870 metrics / second (178700 datapoints / second)

RxJava2:

1 datapoint -> 76216 metrics / second (76216 datapoints / second)

10 datapoints -> 20088 metrics / second (200880 datapoints / second)

HWKMETRICS-599 without retryWhen made no difference (an assumption that 
this was the problem creator).

The 1 datapoint per metric is the most interesting use-case, so that's 
something we should concentrate on. But before someone asks:

100 metrics, 1000 datapoints per metric and batch size of 1000 (instead 
of default 50 in the micro-batching):

298030 datapoints / second.

That's all folks!

   - Micke

On 02/11/2017 12:04 PM, Michael Burman wrote:
> Hi,
>
> I did yesterday evening and today some testing on how using RxJava2
> would benefit us (I'm expecting more from RxJava 2.1 actually, since it
> has some enhanced parallelism features which we might benefit from).
>
> Short notes from RxJava2 migration, it's more painful than I assumed.
> The code changes can be small in terms of lines of code changed, but
> almost every method has had their signature or behavior changed. So at
> least I've had to read the documentation all the time when doing things
> and trying to unlearn what I've done in the RxJava1.
>
> And all this comes with a backwards compatibility pressure for Java 6
> (so you can't benefit from many Java 8 advantages). Reactive-Commons /
> Reactor have started from Java 8 to provide cleaner implementation. Grr.
>
> I wrote a simple write path modification in the PR #762 (metrics) that
> writes Gauges using RxJava2 ported micro-batching feature. There's still
> some RxJavaInterOp use in it, so that might slow down the performance a
> little bit. However, it is possible to merge these two codes. There are
> also some other optimizations I think could be worth it.
>
> I'd advice against it though, reading gets quite complex. I would almost
> suggest that we would do the MetricsServiceImpl/DataAccessImpl merging
> by rewriting small parts at a time in the new class with RxJava2 and
> make that call the old code with RxJavaInterOp. That way we could move
> slowly to the newer codebase.
>
> I fixed the JMH-benchmarks (as they're not compiled in our CI and were
> actually broken by some other PRs) and ran some tests. These are the
> tests that measure only the metrics-core-service performance and do not
> touch the REST-interface (or Wildfly) at all, thus giving better
> comparison in how our internal changes behave.
>
> What I'm seeing is around 20-30% difference in performance when writing
> gauges this way. So this should offset some of the issues we saw when we
> improved error handling (which caused performance degradation). I did
> ran into the HWKMETRICS-542 (BusyPoolException) so the tests were run
> with 1024 connections.
>
> I'll continue next week some more testing, but at the same time I proved
> that the micro-batching features do improve performance in the internal
> processing, especially when there's small amount of writers to a single
> node. But testing those features could probably benefit from more
> benchmark tests without WIldfly (which takes so much processing power
> that most performance improvements can't be measured correctly anymore).
>
>      - Micke
> _______________________________________________
> hawkular-dev mailing list
> hawkular-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/hawkular-dev