[Hawkular-commits] [hawkular/hawkular-metrics] 7b5d80: ETL based compression methods in the core for Gaug...

jsanda jsanda at redhat.com
Tue Oct 4 09:14:27 EDT 2016 

  Branch: refs/heads/master
  Home:   https://github.com/hawkular/hawkular-metrics
  Commit: 7b5d806a7b5c441db5702d2a0f157f80bfd7eae9
      https://github.com/hawkular/hawkular-metrics/commit/7b5d806a7b5c441db5702d2a0f157f80bfd7eae9
  Author: Michael Burman <yak at iki.fi>
  Date:   2016-10-04 (Tue, 04 Oct 2016)

  Changed paths:
    M api/metrics-api-jaxrs/src/main/java/org/hawkular/metrics/api/jaxrs/MetricsServiceLifecycle.java
    M core/datetime-service/src/main/java/org/hawkular/metrics/datetime/DateTimeService.java
    M core/metrics-core-service/pom.xml
    A core/metrics-core-service/src/main/java/org/hawkular/metrics/core/jobs/CompressData.java
    M core/metrics-core-service/src/main/java/org/hawkular/metrics/core/jobs/JobsService.java
    M core/metrics-core-service/src/main/java/org/hawkular/metrics/core/jobs/JobsServiceImpl.java
    M core/metrics-core-service/src/main/java/org/hawkular/metrics/core/service/DataAccess.java
    M core/metrics-core-service/src/main/java/org/hawkular/metrics/core/service/DataAccessImpl.java
    M core/metrics-core-service/src/main/java/org/hawkular/metrics/core/service/MetricsService.java
    M core/metrics-core-service/src/main/java/org/hawkular/metrics/core/service/MetricsServiceImpl.java
    A core/metrics-core-service/src/main/java/org/hawkular/metrics/core/service/compress/CompressedPointContainer.java
    A core/metrics-core-service/src/main/java/org/hawkular/metrics/core/service/compress/CompressorHeader.java
    A core/metrics-core-service/src/main/java/org/hawkular/metrics/core/service/compress/TagsDeserializer.java
    A core/metrics-core-service/src/main/java/org/hawkular/metrics/core/service/compress/TagsSerializer.java
    M core/metrics-core-service/src/main/java/org/hawkular/metrics/core/service/transformers/BatchStatementTransformer.java
    A core/metrics-core-service/src/main/java/org/hawkular/metrics/core/service/transformers/DataPointCompressTransformer.java
    A core/metrics-core-service/src/main/java/org/hawkular/metrics/core/service/transformers/DataPointDecompressTransformer.java
    M core/metrics-core-service/src/main/java/org/hawkular/metrics/core/service/transformers/MetricFromDataRowTransformer.java
    M core/metrics-core-service/src/main/java/org/hawkular/metrics/core/service/transformers/MetricFromFullDataRowTransformer.java
    A core/metrics-core-service/src/test/java/org/hawkular/metrics/core/compress/CompressorHeaderTest.java
    A core/metrics-core-service/src/test/java/org/hawkular/metrics/core/compress/TagsMapSerializingTest.java
    A core/metrics-core-service/src/test/java/org/hawkular/metrics/core/jobs/CompressDataJobITest.java
    M core/metrics-core-service/src/test/java/org/hawkular/metrics/core/jobs/DeleteTenantITest.java
    M core/metrics-core-service/src/test/java/org/hawkular/metrics/core/service/DataAccessITest.java
    M core/metrics-core-service/src/test/java/org/hawkular/metrics/core/service/DelegatingDataAccess.java
    M core/metrics-core-service/src/test/java/org/hawkular/metrics/core/service/metrics/GaugeITest.java
    M core/metrics-model/src/main/java/org/hawkular/metrics/model/AvailabilityType.java
    M core/schema/src/main/java/org/hawkular/metrics/schema/SchemaService.java
    M core/schema/src/main/resources/org/hawkular/schema/cassalog.groovy
    A core/schema/src/main/resources/org/hawkular/schema/updates/schema-0.20.0.groovy
    M integration-tests/rest-tests-jaxrs/src/test/groovy/org/hawkular/metrics/rest/RESTTest.groovy
    M job-scheduler/src/main/java/org/hawkular/metrics/scheduler/api/Scheduler.java
    M job-scheduler/src/main/java/org/hawkular/metrics/scheduler/impl/SchedulerImpl.java
    M job-scheduler/src/main/java/org/hawkular/metrics/scheduler/impl/TestScheduler.java
    M pom.xml

  Log Message:
  -----------
  ETL based compression methods in the core for Gauge data

Add token range queries for metric partition keys

Reorder DISTINCT queries to match the partition key (required in the tokens for example), replace bind(Objects...) with bind().setXXX

Implement the CompressData job and the required changes to the MetricsService

Implement jobServices for ETL compress (not activated yet, needs rebase from master)

Add correct timing functions for the CompressDataJob and add the ability to fetch all jobs from JobScheduler

Add compressor header to every compressed field

Move bindpoint changes to another PR

Fix trigger time calculation to match UTC

Rebased the TestScheduler changes and updated compression library to 1.1.0

Use range delete in addAndCompress, requires removing batch statement as that does not preserve order

Add data_compressed table where all the compressed rows are stored in

Add ability to disable the compression job and remove data locality stuff for another PR..

Add support for compressing COUNTER and AVAILABILITY also

Add Tags serialization/deserialization and schema changes for them

Change CompressData to use jobTriggerTime and the tests to force UTC

Enable findDataPoints to use compressedData for COUNTER & AVAILABILITY and fix DataAccessITest to clean up compressed_table also before test

Improve CompressionHeader functionality

Enable tagsSerialization and test for that


  Commit: 45da8808ecebc478c9ccea4c15b504b1f229d4b6
      https://github.com/hawkular/hawkular-metrics/commit/45da8808ecebc478c9ccea4c15b504b1f229d4b6
  Author: jsanda <jsanda at redhat.com>
  Date:   2016-10-04 (Tue, 04 Oct 2016)

  Changed paths:
    M api/metrics-api-jaxrs/src/main/java/org/hawkular/metrics/api/jaxrs/MetricsServiceLifecycle.java
    M core/datetime-service/src/main/java/org/hawkular/metrics/datetime/DateTimeService.java
    M core/metrics-core-service/pom.xml
    A core/metrics-core-service/src/main/java/org/hawkular/metrics/core/jobs/CompressData.java
    M core/metrics-core-service/src/main/java/org/hawkular/metrics/core/jobs/JobsService.java
    M core/metrics-core-service/src/main/java/org/hawkular/metrics/core/jobs/JobsServiceImpl.java
    M core/metrics-core-service/src/main/java/org/hawkular/metrics/core/service/DataAccess.java
    M core/metrics-core-service/src/main/java/org/hawkular/metrics/core/service/DataAccessImpl.java
    M core/metrics-core-service/src/main/java/org/hawkular/metrics/core/service/MetricsService.java
    M core/metrics-core-service/src/main/java/org/hawkular/metrics/core/service/MetricsServiceImpl.java
    A core/metrics-core-service/src/main/java/org/hawkular/metrics/core/service/compress/CompressedPointContainer.java
    A core/metrics-core-service/src/main/java/org/hawkular/metrics/core/service/compress/CompressorHeader.java
    A core/metrics-core-service/src/main/java/org/hawkular/metrics/core/service/compress/TagsDeserializer.java
    A core/metrics-core-service/src/main/java/org/hawkular/metrics/core/service/compress/TagsSerializer.java
    M core/metrics-core-service/src/main/java/org/hawkular/metrics/core/service/transformers/BatchStatementTransformer.java
    A core/metrics-core-service/src/main/java/org/hawkular/metrics/core/service/transformers/DataPointCompressTransformer.java
    A core/metrics-core-service/src/main/java/org/hawkular/metrics/core/service/transformers/DataPointDecompressTransformer.java
    M core/metrics-core-service/src/main/java/org/hawkular/metrics/core/service/transformers/MetricFromDataRowTransformer.java
    M core/metrics-core-service/src/main/java/org/hawkular/metrics/core/service/transformers/MetricFromFullDataRowTransformer.java
    A core/metrics-core-service/src/test/java/org/hawkular/metrics/core/compress/CompressorHeaderTest.java
    A core/metrics-core-service/src/test/java/org/hawkular/metrics/core/compress/TagsMapSerializingTest.java
    A core/metrics-core-service/src/test/java/org/hawkular/metrics/core/jobs/CompressDataJobITest.java
    M core/metrics-core-service/src/test/java/org/hawkular/metrics/core/jobs/DeleteTenantITest.java
    M core/metrics-core-service/src/test/java/org/hawkular/metrics/core/service/DataAccessITest.java
    M core/metrics-core-service/src/test/java/org/hawkular/metrics/core/service/DelegatingDataAccess.java
    M core/metrics-core-service/src/test/java/org/hawkular/metrics/core/service/metrics/GaugeITest.java
    M core/metrics-model/src/main/java/org/hawkular/metrics/model/AvailabilityType.java
    M core/schema/src/main/java/org/hawkular/metrics/schema/SchemaService.java
    M core/schema/src/main/resources/org/hawkular/schema/cassalog.groovy
    A core/schema/src/main/resources/org/hawkular/schema/updates/schema-0.20.0.groovy
    M integration-tests/rest-tests-jaxrs/src/test/groovy/org/hawkular/metrics/rest/RESTTest.groovy
    M job-scheduler/src/main/java/org/hawkular/metrics/scheduler/api/Scheduler.java
    M job-scheduler/src/main/java/org/hawkular/metrics/scheduler/impl/SchedulerImpl.java
    M job-scheduler/src/main/java/org/hawkular/metrics/scheduler/impl/TestScheduler.java
    M pom.xml

  Log Message:
  -----------
  Merge pull request #595 from burmanm/etl_compression

ETL compression process based on Gorilla algorithms


Compare: https://github.com/hawkular/hawkular-metrics/compare/e08798032910...45da8808eceb

More information about the hawkular-commits mailing list