[Hawkular-dev] [Metrics] Evaluating performance of #groupBy vs #window for stats computation
Thomas Segismont
tsegismo at redhat.com
Thu Aug 27 10:15:13 EDT 2015
Hi,
Recently, we've ported to RxJava the stats computation code, a part not
fully "reactive" yet. The intention was to be able to compute statistics
on raw data for large period of times, with low requirements on memory
(the initial implementation needed all data loaded at once in a list).
If you're interested in the $subject and don't known what stats
computation is, it's an API metrics provides to give the user stats
(min/max/avg/median/95th percentile) on raw data. The stats are given in
buckets (portions of time). As an example, it's the API you use if you
want monthly stats on a year of data, or hourly stats for a day, ... etc.
The implementation makes use of the Observable#groupBy operator. This
allows to determine the bucket each data point belongs to, and then we
#collect the points in each bucket to compute the stats.
https://github.com/hawkular/hawkular-metrics/blob/master/core/metrics-core-impl/src/main/java/org/hawkular/metrics/core/impl/MetricsServiceImpl.java#L803-L813
As we were talking on a related topic (stats for sliding windows), John
wondered if a solution based on Observable#window would perform equally.
So I've implemented this solution and instrumented the code to determine
where the system was spending the execution time. Then I tried both
solutions with different bucket sizes, metric resolutions, numbers of
buckets.
Here are my observations.
#groupBy and #window behave about the same in our case (I've tried with
a few buckets up to a thousand buckets)
Response time increases linearly with the number of data points loaded
from the C*. Response time here means time elapsed between the moment
Wildfly invokes the JAX-RS handler method and we resume the AsyncResponse.
The bottleneck is the data loading and mapping: 95% of execution time.
C* Row to object mapping is nearly 50% of it.
And my conclusions.
When working with non overlapping buckets (not sliding windows),
readability should determine the use of #groupBy or #window, not
performance.
It seems possible to get a better response time. Currently we transform
a ResultSet into an Observable<Row> with a simple call to
Observable#from (ResultSet is an Iterable<Row>). By default, the C*
driver fetches a page of data, and only when it's entirely consumed,
fetches the following one. But ResultSet has a #fetchMoreResults method
which we could use to fetch pages ahead of time (see
http://docs.datastax.com/en/drivers/java/2.0/com/datastax/driver/core/ResultSet.html).
Then, while Rx computation threads would spend time on mapping a Row to
a DataPoint<T>, the C* driver could load more Rows.
Attached are:
- a patch of the changes implemented for testing
- some of the results
Regards,
Thomas
-------------- next part --------------
int nbOfBuckets = 100
long bucketSize = Duration.standardHours(1).millis
int interval = Duration.standardSeconds(1).millis
12:24:41,156 ERROR [org.hawkular.metrics.core.impl.MetricsServiceImpl] (RxComputationThreadPool-2) mapping time = 1055 ms
12:24:41,156 ERROR [org.hawkular.metrics.api.jaxrs.util.ApiUtils] (RxComputationThreadPool-2) reponse creation time = 19,29 μs
12:24:41,157 ERROR [org.hawkular.metrics.api.jaxrs.handler.GaugeHandler] (RxComputationThreadPool-2) grand total = 2,147 s
int nbOfBuckets = 500
long bucketSize = Duration.standardHours(1).millis
int interval = Duration.standardSeconds(1).millis
12:29:02,969 ERROR [org.hawkular.metrics.core.impl.MetricsServiceImpl] (RxComputationThreadPool-3) mapping time = 5095 ms
12:29:02,969 ERROR [org.hawkular.metrics.api.jaxrs.util.ApiUtils] (RxComputationThreadPool-3) reponse creation time = 15,20 μs
12:29:02,978 ERROR [org.hawkular.metrics.api.jaxrs.handler.GaugeHandler] (RxComputationThreadPool-3) grand total = 12,53 s
int nbOfBuckets = 360
long bucketSize = Duration.standardHours(24).millis
int interval = Duration.standardSeconds(30).millis
12:34:17,860 ERROR [org.hawkular.metrics.core.impl.MetricsServiceImpl] (RxComputationThreadPool-4) mapping time = 2835 ms
12:34:17,860 ERROR [org.hawkular.metrics.api.jaxrs.util.ApiUtils] (RxComputationThreadPool-4) reponse creation time = 19,30 μs
12:34:17,863 ERROR [org.hawkular.metrics.api.jaxrs.handler.GaugeHandler] (RxComputationThreadPool-4) grand total = 6,499 s
-------------- next part --------------
A non-text attachment was scrubbed...
Name: getXXXStats_Perf.patch
Type: text/x-patch
Size: 13612 bytes
Desc: not available
Url : http://lists.jboss.org/pipermail/hawkular-dev/attachments/20150827/d7e419d4/attachment-0001.bin
-------------- next part --------------
int nbOfBuckets = 100
long bucketSize = Duration.standardHours(1).millis
int interval = Duration.standardSeconds(1).millis
11:10:49,430 ERROR [org.hawkular.metrics.core.impl.MetricsServiceImpl] (RxComputationThreadPool-2) loading = 815,3 ms
11:10:51,124 ERROR [org.hawkular.metrics.core.impl.MetricsServiceImpl] (RxComputationThreadPool-2) mapping time = 1634 ms
11:10:51,144 ERROR [org.hawkular.metrics.core.impl.MetricsServiceImpl] (default task-27) toList = 2,548 s
11:10:51,303 ERROR [org.hawkular.metrics.api.jaxrs.util.ApiUtils] (default task-27) reponse creation time = 24,56 μs
11:10:51,338 ERROR [org.hawkular.metrics.core.impl.MetricsServiceImpl] (default task-27) rx = 194,1 ms
11:10:51,339 ERROR [org.hawkular.metrics.api.jaxrs.handler.GaugeHandler] (default task-27) grand total = 2,743 s
int nbOfBuckets = 500
long bucketSize = Duration.standardHours(1).millis
int interval = Duration.standardSeconds(1).millis
11:12:37,200 ERROR [org.hawkular.metrics.core.impl.MetricsServiceImpl] (RxComputationThreadPool-3) loading = 7,055 s
11:12:44,303 ERROR [org.hawkular.metrics.core.impl.MetricsServiceImpl] (RxComputationThreadPool-3) mapping time = 6837 ms
11:12:44,352 ERROR [org.hawkular.metrics.core.impl.MetricsServiceImpl] (default task-4) toList = 14,27 s
11:12:45,056 ERROR [org.hawkular.metrics.api.jaxrs.util.ApiUtils] (default task-4) reponse creation time = 28,98 μs
11:12:45,067 ERROR [org.hawkular.metrics.core.impl.MetricsServiceImpl] (default task-4) rx = 714,8 ms
11:12:45,068 ERROR [org.hawkular.metrics.api.jaxrs.handler.GaugeHandler] (default task-4) grand total = 14,98 s
int nbOfBuckets = 360
long bucketSize = Duration.standardHours(24).millis
int interval = Duration.standardSeconds(30).millis
11:16:39,241 ERROR [org.hawkular.metrics.core.impl.MetricsServiceImpl] (RxComputationThreadPool-4) loading = 2,839 s
11:16:43,662 ERROR [org.hawkular.metrics.core.impl.MetricsServiceImpl] (RxComputationThreadPool-4) mapping time = 4258 ms
11:16:43,697 ERROR [org.hawkular.metrics.core.impl.MetricsServiceImpl] (default task-21) toList = 7,314 s
11:16:44,137 ERROR [org.hawkular.metrics.api.jaxrs.util.ApiUtils] (default task-21) reponse creation time = 44,73 μs
11:16:44,141 ERROR [org.hawkular.metrics.core.impl.MetricsServiceImpl] (default task-21) rx = 443,7 ms
11:16:44,141 ERROR [org.hawkular.metrics.api.jaxrs.handler.GaugeHandler] (default task-21) grand total = 7,758 s
-------------- next part --------------
int nbOfBuckets = 100
long bucketSize = Duration.standardHours(1).millis
int interval = Duration.standardSeconds(1).millis
12:15:13,147 ERROR [org.hawkular.metrics.core.impl.MetricsServiceImpl] (RxComputationThreadPool-2) loading = 971,0 ms
12:15:14,274 ERROR [org.hawkular.metrics.core.impl.MetricsServiceImpl] (RxComputationThreadPool-2) mapping time = 942 ms
12:15:14,292 ERROR [org.hawkular.metrics.core.impl.MetricsServiceImpl] (default task-56) toList = 2,130 s
12:15:14,436 ERROR [org.hawkular.metrics.api.jaxrs.util.ApiUtils] (default task-56) reponse creation time = 20,35 μs
12:15:14,437 ERROR [org.hawkular.metrics.core.impl.MetricsServiceImpl] (default task-56) rx = 144,9 ms
12:15:14,437 ERROR [org.hawkular.metrics.api.jaxrs.handler.GaugeHandler] (default task-56) grand total = 2,275 s
int nbOfBuckets = 500
long bucketSize = Duration.standardHours(1).millis
int interval = Duration.standardSeconds(1).millis
12:17:26,538 ERROR [org.hawkular.metrics.core.impl.MetricsServiceImpl] (RxComputationThreadPool-3) loading = 8,138 s
12:17:35,229 ERROR [org.hawkular.metrics.core.impl.MetricsServiceImpl] (RxComputationThreadPool-3) mapping time = 8088 ms
12:17:35,287 ERROR [org.hawkular.metrics.core.impl.MetricsServiceImpl] (default task-33) toList = 16,95 s
12:17:36,009 ERROR [org.hawkular.metrics.api.jaxrs.util.ApiUtils] (default task-33) reponse creation time = 17,14 μs
12:17:36,013 ERROR [org.hawkular.metrics.core.impl.MetricsServiceImpl] (default task-33) rx = 726,4 ms
12:17:36,014 ERROR [org.hawkular.metrics.api.jaxrs.handler.GaugeHandler] (default task-33) grand total = 17,68 s
int nbOfBuckets = 360
long bucketSize = Duration.standardHours(24).millis
int interval = Duration.standardSeconds(30).millis
12:19:40,728 ERROR [org.hawkular.metrics.core.impl.MetricsServiceImpl] (RxComputationThreadPool-4) loading = 2,781 s
12:19:43,707 ERROR [org.hawkular.metrics.core.impl.MetricsServiceImpl] (RxComputationThreadPool-4) mapping time = 2746 ms
12:19:43,737 ERROR [org.hawkular.metrics.core.impl.MetricsServiceImpl] (default task-50) toList = 5,809 s
12:19:44,075 ERROR [org.hawkular.metrics.api.jaxrs.util.ApiUtils] (default task-50) reponse creation time = 16,62 μs
12:19:44,082 ERROR [org.hawkular.metrics.core.impl.MetricsServiceImpl] (default task-50) rx = 344,5 ms
12:19:44,082 ERROR [org.hawkular.metrics.api.jaxrs.handler.GaugeHandler] (default task-50) grand total = 6,154 s
More information about the hawkular-dev
mailing list