[Hawkular-commits] [hawkular/hawkular-metrics] fba9ff: Implementation of statistics computation based on ...

[jsanda]

  Branch: refs/heads/master
  Home:   https://github.com/hawkular/hawkular-metrics
  Commit: fba9fff2ff38eb79697410fbcf966f9ac2899d3c
      https://github.com/hawkular/hawkular-metrics/commit/fba9fff2ff38eb79697410fbcf966f9ac2899d3c
  Author: Thomas Segismont <tsegismo at redhat.com>
  Date:   2015-08-07 (Fri, 07 Aug 2015)

  Changed paths:
    M api/metrics-api-jaxrs/src/main/java/org/hawkular/metrics/api/jaxrs/handler/AvailabilityHandler.java
    M api/metrics-api-jaxrs/src/main/java/org/hawkular/metrics/api/jaxrs/handler/GaugeHandler.java
    M api/metrics-api-jaxrs/src/main/java/org/hawkular/metrics/api/jaxrs/util/ApiUtils.java
    R core/metrics-core-api/src/main/java/org/hawkular/metrics/core/api/AvailabilityBucketDataPoint.java
    A core/metrics-core-api/src/main/java/org/hawkular/metrics/core/api/AvailabilityBucketPoint.java
    A core/metrics-core-api/src/main/java/org/hawkular/metrics/core/api/BucketPoint.java
    R core/metrics-core-api/src/main/java/org/hawkular/metrics/core/api/BucketedOutput.java
    M core/metrics-core-api/src/main/java/org/hawkular/metrics/core/api/Buckets.java
    R core/metrics-core-api/src/main/java/org/hawkular/metrics/core/api/GaugeBucketDataPoint.java
    A core/metrics-core-api/src/main/java/org/hawkular/metrics/core/api/GaugeBucketPoint.java
    M core/metrics-core-api/src/main/java/org/hawkular/metrics/core/api/MetricsService.java
    R core/metrics-core-impl/src/main/java/org/hawkular/metrics/core/impl/AvailabilityBucketedOutputMapper.java
    A core/metrics-core-impl/src/main/java/org/hawkular/metrics/core/impl/AvailabilityDataPointCollector.java
    R core/metrics-core-impl/src/main/java/org/hawkular/metrics/core/impl/BucketedOutputMapper.java
    R core/metrics-core-impl/src/main/java/org/hawkular/metrics/core/impl/GaugeBucketedOutputMapper.java
    A core/metrics-core-impl/src/main/java/org/hawkular/metrics/core/impl/GaugeDataPointCollector.java
    M core/metrics-core-impl/src/main/java/org/hawkular/metrics/core/impl/MetricsServiceImpl.java
    R core/metrics-core-impl/src/test/java/org/hawkular/metrics/core/impl/AvailabilityBucketDataPointMatcher.java
    A core/metrics-core-impl/src/test/java/org/hawkular/metrics/core/impl/AvailabilityBucketPointMatcher.java
    R core/metrics-core-impl/src/test/java/org/hawkular/metrics/core/impl/AvailabilityBucketedOutputMapperTest.java
    A core/metrics-core-impl/src/test/java/org/hawkular/metrics/core/impl/AvailabilityDataPointCollectorTest.java
    M rest-tests/src/test/groovy/org/hawkular/metrics/rest/GaugeMetricStatisticsITest.groovy
    M rest-tests/src/test/groovy/org/hawkular/metrics/rest/RESTTest.groovy

  Log Message:
  -----------
  Implementation of statistics computation based on RxJava

This is a combination of 8 commits.

The first commit's message is:
Introduce BucketedOutputOperator and children

This is the 2nd commit message:

Update MetricsServiceImpl to use BucketedOutputOperator

This is the 3rd commit message:

Gauges buckets computation without custom operator

This is to show what the process could look like, code for avail has not
been updated, and code for counters not implemented.

In this attempt, no custom operator is used. Instead, N(=number of
buckets) requests for data are sent in parallel. Then the resulting
observables are flat mapped to form an observable of bucket points.

The bucket point observable emits just one value. The value is created
by processing data points as they arrive, instead of buffering
all the points for the bucket in an array.
This has a huge implication on stats computation. Some stats algos are
"storeless", which means you can get the exact result without buffering
the data. Examples are min, max, avg. But some are not, like percentile.

As a consequence, for median and 95th percentile, we need to use
an approximative algorithm, provided by
org.apache.commons.math3.stat.descriptive.rank.PSquarePercentile.
See http://www.cse.wustl.edu/~jain/papers/psqr.htm for a description of
the algorithm.
On the upside, this algorithm allows to compute stats when the number of
data points in the bucket is VERY large, with low memory consumption.
On the downside, the result might look a bit inaccurate when there VERY
few points in the buckets. It's also sensible to order of value
submission.

Note that the observable of bucket points will often be incorrectly
ordered, as there's no guarantee in the order the responses will be processed.

Side work:
- log throwable at TRACE level for 500 responses
- overloaded assertDoubleEquals so that caller can pass a failure
  message

This is the 4th commit message:

Refactor to use #collect for readability

This is the 5th commit message:

Refactor to use  as single request to C*

Sending N requests to C* might be bad wrt performance, especially if N(=number of buckets) is high

This implementation uses #groupBy. Not sure how to make that work with #window

This is the 6th commit message:

Remove unused code

This is the 7th commit message:

Apply logic to avail metrics

This is the 8th commit message:

Final step, refactoring for better readability


  Commit: c0299235d137ee950971e1feeb055ff2cc5a8968
      https://github.com/hawkular/hawkular-metrics/commit/c0299235d137ee950971e1feeb055ff2cc5a8968
  Author: jsanda <jsanda at redhat.com>
  Date:   2015-08-10 (Mon, 10 Aug 2015)

  Changed paths:
    M api/metrics-api-jaxrs/src/main/java/org/hawkular/metrics/api/jaxrs/handler/AvailabilityHandler.java
    M api/metrics-api-jaxrs/src/main/java/org/hawkular/metrics/api/jaxrs/handler/GaugeHandler.java
    M api/metrics-api-jaxrs/src/main/java/org/hawkular/metrics/api/jaxrs/util/ApiUtils.java
    R core/metrics-core-api/src/main/java/org/hawkular/metrics/core/api/AvailabilityBucketDataPoint.java
    A core/metrics-core-api/src/main/java/org/hawkular/metrics/core/api/AvailabilityBucketPoint.java
    A core/metrics-core-api/src/main/java/org/hawkular/metrics/core/api/BucketPoint.java
    R core/metrics-core-api/src/main/java/org/hawkular/metrics/core/api/BucketedOutput.java
    M core/metrics-core-api/src/main/java/org/hawkular/metrics/core/api/Buckets.java
    R core/metrics-core-api/src/main/java/org/hawkular/metrics/core/api/GaugeBucketDataPoint.java
    A core/metrics-core-api/src/main/java/org/hawkular/metrics/core/api/GaugeBucketPoint.java
    M core/metrics-core-api/src/main/java/org/hawkular/metrics/core/api/MetricsService.java
    R core/metrics-core-impl/src/main/java/org/hawkular/metrics/core/impl/AvailabilityBucketedOutputMapper.java
    A core/metrics-core-impl/src/main/java/org/hawkular/metrics/core/impl/AvailabilityDataPointCollector.java
    R core/metrics-core-impl/src/main/java/org/hawkular/metrics/core/impl/BucketedOutputMapper.java
    R core/metrics-core-impl/src/main/java/org/hawkular/metrics/core/impl/GaugeBucketedOutputMapper.java
    A core/metrics-core-impl/src/main/java/org/hawkular/metrics/core/impl/GaugeDataPointCollector.java
    M core/metrics-core-impl/src/main/java/org/hawkular/metrics/core/impl/MetricsServiceImpl.java
    R core/metrics-core-impl/src/test/java/org/hawkular/metrics/core/impl/AvailabilityBucketDataPointMatcher.java
    A core/metrics-core-impl/src/test/java/org/hawkular/metrics/core/impl/AvailabilityBucketPointMatcher.java
    R core/metrics-core-impl/src/test/java/org/hawkular/metrics/core/impl/AvailabilityBucketedOutputMapperTest.java
    A core/metrics-core-impl/src/test/java/org/hawkular/metrics/core/impl/AvailabilityDataPointCollectorTest.java
    M rest-tests/src/test/groovy/org/hawkular/metrics/rest/GaugeMetricStatisticsITest.groovy
    M rest-tests/src/test/groovy/org/hawkular/metrics/rest/RESTTest.groovy

  Log Message:
  -----------
  Merge pull request #302 from tsegismont/jira/HWKMETRICS-206

HWKMETRICS-206 Implementation of statistics computation based on RxJava


Compare: https://github.com/hawkular/hawkular-metrics/compare/38ff28cd565b...c0299235d137