[Hawkular-commits] [hawkular/hawkular-metrics] fba9ff: Implementation of statistics computation based on ...

Branch: refs/heads/master Home: https://github.com/hawkular/hawkular-metrics Commit: fba9fff2ff38eb79697410fbcf966f9ac2899d3c https://github.com/hawkular/hawkular-metrics/commit/fba9fff2ff38eb7969741... Author: Thomas Segismont <tsegismo(a)redhat.com&gt; Date: 2015-08-07 (Fri, 07 Aug 2015) Changed paths: M api/metrics-api-jaxrs/src/main/java/org/hawkular/metrics/api/jaxrs/handler/AvailabilityHandler.java M api/metrics-api-jaxrs/src/main/java/org/hawkular/metrics/api/jaxrs/handler/GaugeHandler.java M api/metrics-api-jaxrs/src/main/java/org/hawkular/metrics/api/jaxrs/util/ApiUtils.java R core/metrics-core-api/src/main/java/org/hawkular/metrics/core/api/AvailabilityBucketDataPoint.java A core/metrics-core-api/src/main/java/org/hawkular/metrics/core/api/AvailabilityBucketPoint.java A core/metrics-core-api/src/main/java/org/hawkular/metrics/core/api/BucketPoint.java R core/metrics-core-api/src/main/java/org/hawkular/metrics/core/api/BucketedOutput.java M core/metrics-core-api/src/main/java/org/hawkular/metrics/core/api/Buckets.java R core/metrics-core-api/src/main/java/org/hawkular/metrics/core/api/GaugeBucketDataPoint.java A core/metrics-core-api/src/main/java/org/hawkular/metrics/core/api/GaugeBucketPoint.java M core/metrics-core-api/src/main/java/org/hawkular/metrics/core/api/MetricsService.java R core/metrics-core-impl/src/main/java/org/hawkular/metrics/core/impl/AvailabilityBucketedOutputMapper.java A core/metrics-core-impl/src/main/java/org/hawkular/metrics/core/impl/AvailabilityDataPointCollector.java R core/metrics-core-impl/src/main/java/org/hawkular/metrics/core/impl/BucketedOutputMapper.java R core/metrics-core-impl/src/main/java/org/hawkular/metrics/core/impl/GaugeBucketedOutputMapper.java A core/metrics-core-impl/src/main/java/org/hawkular/metrics/core/impl/GaugeDataPointCollector.java M core/metrics-core-impl/src/main/java/org/hawkular/metrics/core/impl/MetricsServiceImpl.java R core/metrics-core-impl/src/test/java/org/hawkular/metrics/core/impl/AvailabilityBucketDataPointMatcher.java A core/metrics-core-impl/src/test/java/org/hawkular/metrics/core/impl/AvailabilityBucketPointMatcher.java R core/metrics-core-impl/src/test/java/org/hawkular/metrics/core/impl/AvailabilityBucketedOutputMapperTest.java A core/metrics-core-impl/src/test/java/org/hawkular/metrics/core/impl/AvailabilityDataPointCollectorTest.java M rest-tests/src/test/groovy/org/hawkular/metrics/rest/GaugeMetricStatisticsITest.groovy M rest-tests/src/test/groovy/org/hawkular/metrics/rest/RESTTest.groovy Log Message: ----------- Implementation of statistics computation based on RxJava This is a combination of 8 commits. The first commit's message is: Introduce BucketedOutputOperator and children This is the 2nd commit message: Update MetricsServiceImpl to use BucketedOutputOperator This is the 3rd commit message: Gauges buckets computation without custom operator This is to show what the process could look like, code for avail has not been updated, and code for counters not implemented. In this attempt, no custom operator is used. Instead, N(=number of buckets) requests for data are sent in parallel. Then the resulting observables are flat mapped to form an observable of bucket points. The bucket point observable emits just one value. The value is created by processing data points as they arrive, instead of buffering all the points for the bucket in an array. This has a huge implication on stats computation. Some stats algos are "storeless", which means you can get the exact result without buffering the data. Examples are min, max, avg. But some are not, like percentile. As a consequence, for median and 95th percentile, we need to use an approximative algorithm, provided by org.apache.commons.math3.stat.descriptive.rank.PSquarePercentile. See http://www.cse.wustl.edu/~jain/papers/psqr.htm for a description of the algorithm. On the upside, this algorithm allows to compute stats when the number of data points in the bucket is VERY large, with low memory consumption. On the downside, the result might look a bit inaccurate when there VERY few points in the buckets. It's also sensible to order of value submission. Note that the observable of bucket points will often be incorrectly ordered, as there's no guarantee in the order the responses will be processed. Side work: - log throwable at TRACE level for 500 responses - overloaded assertDoubleEquals so that caller can pass a failure message This is the 4th commit message: Refactor to use #collect for readability This is the 5th commit message: Refactor to use as single request to C* Sending N requests to C* might be bad wrt performance, especially if N(=number of buckets) is high This implementation uses #groupBy. Not sure how to make that work with #window This is the 6th commit message: Remove unused code This is the 7th commit message: Apply logic to avail metrics This is the 8th commit message: Final step, refactoring for better readability Commit: c0299235d137ee950971e1feeb055ff2cc5a8968 https://github.com/hawkular/hawkular-metrics/commit/c0299235d137ee950971e... Author: jsanda <jsanda(a)redhat.com&gt; Date: 2015-08-10 (Mon, 10 Aug 2015) Changed paths: M api/metrics-api-jaxrs/src/main/java/org/hawkular/metrics/api/jaxrs/handler/AvailabilityHandler.java M api/metrics-api-jaxrs/src/main/java/org/hawkular/metrics/api/jaxrs/handler/GaugeHandler.java M api/metrics-api-jaxrs/src/main/java/org/hawkular/metrics/api/jaxrs/util/ApiUtils.java R core/metrics-core-api/src/main/java/org/hawkular/metrics/core/api/AvailabilityBucketDataPoint.java A core/metrics-core-api/src/main/java/org/hawkular/metrics/core/api/AvailabilityBucketPoint.java A core/metrics-core-api/src/main/java/org/hawkular/metrics/core/api/BucketPoint.java R core/metrics-core-api/src/main/java/org/hawkular/metrics/core/api/BucketedOutput.java M core/metrics-core-api/src/main/java/org/hawkular/metrics/core/api/Buckets.java R core/metrics-core-api/src/main/java/org/hawkular/metrics/core/api/GaugeBucketDataPoint.java A core/metrics-core-api/src/main/java/org/hawkular/metrics/core/api/GaugeBucketPoint.java M core/metrics-core-api/src/main/java/org/hawkular/metrics/core/api/MetricsService.java R core/metrics-core-impl/src/main/java/org/hawkular/metrics/core/impl/AvailabilityBucketedOutputMapper.java A core/metrics-core-impl/src/main/java/org/hawkular/metrics/core/impl/AvailabilityDataPointCollector.java R core/metrics-core-impl/src/main/java/org/hawkular/metrics/core/impl/BucketedOutputMapper.java R core/metrics-core-impl/src/main/java/org/hawkular/metrics/core/impl/GaugeBucketedOutputMapper.java A core/metrics-core-impl/src/main/java/org/hawkular/metrics/core/impl/GaugeDataPointCollector.java M core/metrics-core-impl/src/main/java/org/hawkular/metrics/core/impl/MetricsServiceImpl.java R core/metrics-core-impl/src/test/java/org/hawkular/metrics/core/impl/AvailabilityBucketDataPointMatcher.java A core/metrics-core-impl/src/test/java/org/hawkular/metrics/core/impl/AvailabilityBucketPointMatcher.java R core/metrics-core-impl/src/test/java/org/hawkular/metrics/core/impl/AvailabilityBucketedOutputMapperTest.java A core/metrics-core-impl/src/test/java/org/hawkular/metrics/core/impl/AvailabilityDataPointCollectorTest.java M rest-tests/src/test/groovy/org/hawkular/metrics/rest/GaugeMetricStatisticsITest.groovy M rest-tests/src/test/groovy/org/hawkular/metrics/rest/RESTTest.groovy Log Message: ----------- Merge pull request #302 from tsegismont/jira/HWKMETRICS-206 HWKMETRICS-206 Implementation of statistics computation based on RxJava Compare: https://github.com/hawkular/hawkular-metrics/compare/38ff28cd565b...c0299...