9:40 a.m.
Hi,
Testing revealed something else that worried me, I was quite sure that
I've had far higher numbers before than what we have now, so I went back
to test some older versions to see if this was true..
The testing setup is as follows: 4 cores, 8GB, ccm Cassandra (3.0.10,
3.10 made no significant difference here). So all the CPU freed from
HWKMETRICS is used by Cassandra (they compete on resources). Uses
core-metrics-service only, no REST interface - directly writing with
RxJava using addDatapoints.
1 or 10 datapoint per metric, writing 100 000 metrics in one call to
addDatapoints (insertBenchmark, not the -Single ending one). Sources in
the jmh-benchmark module (fix_jmh branch, PR #
0.19.3:
1 datapoint -> 31004 metrics / second (31004 datapoints per second)
10 datapoints -> 19027 metrics / second (190270 datapoints per second)
Current master:
1 datapoint -> 8535 metrics / second (8535 datapoints per second)
10 datapoints -> 7065 metrics / second (70650 datapoints per second)
So performance has dropped significantly between current master and
0.19.3 (0.19.0 was the first release without double writing).
With HWKMETRICS-599 (micro-batching on top of the current master):
RxJava1:
1 datapoint -> 55036 metrics / second (55036 datapoints / second)
10 datapoints -> 17870 metrics / second (178700 datapoints / second)
RxJava2:
1 datapoint -> 76216 metrics / second (76216 datapoints / second)
10 datapoints -> 20088 metrics / second (200880 datapoints / second)
HWKMETRICS-599 without retryWhen made no difference (an assumption that
this was the problem creator).
The 1 datapoint per metric is the most interesting use-case, so that's
something we should concentrate on. But before someone asks:
100 metrics, 1000 datapoints per metric and batch size of 1000 (instead
of default 50 in the micro-batching):
298030 datapoints / second.
That's all folks!
- Micke
On 02/11/2017 12:04 PM, Michael Burman wrote:
Hi,
I did yesterday evening and today some testing on how using RxJava2
would benefit us (I'm expecting more from RxJava 2.1 actually, since it
has some enhanced parallelism features which we might benefit from).
Short notes from RxJava2 migration, it's more painful than I assumed.
The code changes can be small in terms of lines of code changed, but
almost every method has had their signature or behavior changed. So at
least I've had to read the documentation all the time when doing things
and trying to unlearn what I've done in the RxJava1.
And all this comes with a backwards compatibility pressure for Java 6
(so you can't benefit from many Java 8 advantages). Reactive-Commons /
Reactor have started from Java 8 to provide cleaner implementation. Grr.
I wrote a simple write path modification in the PR #762 (metrics) that
writes Gauges using RxJava2 ported micro-batching feature. There's still
some RxJavaInterOp use in it, so that might slow down the performance a
little bit. However, it is possible to merge these two codes. There are
also some other optimizations I think could be worth it.
I'd advice against it though, reading gets quite complex. I would almost
suggest that we would do the MetricsServiceImpl/DataAccessImpl merging
by rewriting small parts at a time in the new class with RxJava2 and
make that call the old code with RxJavaInterOp. That way we could move
slowly to the newer codebase.
I fixed the JMH-benchmarks (as they're not compiled in our CI and were
actually broken by some other PRs) and ran some tests. These are the
tests that measure only the metrics-core-service performance and do not
touch the REST-interface (or Wildfly) at all, thus giving better
comparison in how our internal changes behave.
What I'm seeing is around 20-30% difference in performance when writing
gauges this way. So this should offset some of the issues we saw when we
improved error handling (which caused performance degradation). I did
ran into the HWKMETRICS-542 (BusyPoolException) so the tests were run
with 1024 connections.
I'll continue next week some more testing, but at the same time I proved
that the micro-batching features do improve performance in the internal
processing, especially when there's small amount of writers to a single
node. But testing those features could probably benefit from more
benchmark tests without WIldfly (which takes so much processing power
that most performance improvements can't be measured correctly anymore).
- Micke
_______________________________________________
hawkular-dev mailing list
hawkular-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/hawkular-dev