[Hawkular-commits] [hawkular/hawkular-metrics] b3e276: [HWKMETRICS-168] initial commit for new scheduler ...

Thu Aug 6 09:36:02 EDT 2015

  Branch: refs/heads/task-scheduler
  Home:   https://github.com/hawkular/hawkular-metrics
  Commit: b3e276fc8210dcc07c6a7fe222ae57a217d6ac0d
      https://github.com/hawkular/hawkular-metrics/commit/b3e276fc8210dcc07c6a7fe222ae57a217d6ac0d
  Author: John Sanda <jsanda at redhat.com>
  Date:   2015-08-06 (Thu, 06 Aug 2015)

  Changed paths:
    M schema-manager/src/main/resources/schema.cql
    A task-queue/src/main/java/org/hawkular/metrics/tasks/api/RepeatingTrigger.java
    A task-queue/src/main/java/org/hawkular/metrics/tasks/api/Task2.java
    A task-queue/src/main/java/org/hawkular/metrics/tasks/api/TaskScheduler.java
    M task-queue/src/main/java/org/hawkular/metrics/tasks/impl/Queries.java
    A task-queue/src/main/java/org/hawkular/metrics/tasks/impl/Task2Impl.java
    A task-queue/src/main/java/org/hawkular/metrics/tasks/impl/TaskSchedulerImpl.java
    M task-queue/src/test/java/org/hawkular/metrics/tasks/BaseITest.java
    A task-queue/src/test/java/org/hawkular/metrics/tasks/impl/TaskSchedulerImplTest.java

  Log Message:
  -----------
  [HWKMETRICS-168] initial commit for new scheduler and task classes

  Commit: 6e55547aa3a3279fd325f55b7a0c1ae4458109c2
      https://github.com/hawkular/hawkular-metrics/commit/6e55547aa3a3279fd325f55b7a0c1ae4458109c2
  Author: John Sanda <jsanda at redhat.com>
  Date:   2015-08-06 (Thu, 06 Aug 2015)

  Changed paths:
    M schema-manager/src/main/resources/schema.cql
    M task-queue/src/main/java/org/hawkular/metrics/tasks/api/RepeatingTrigger.java
    M task-queue/src/main/java/org/hawkular/metrics/tasks/api/Task2.java
    M task-queue/src/main/java/org/hawkular/metrics/tasks/api/TaskScheduler.java
    A task-queue/src/main/java/org/hawkular/metrics/tasks/api/Trigger.java
    M task-queue/src/main/java/org/hawkular/metrics/tasks/impl/Queries.java
    M task-queue/src/main/java/org/hawkular/metrics/tasks/impl/Task2Impl.java
    M task-queue/src/main/java/org/hawkular/metrics/tasks/impl/TaskSchedulerImpl.java
    M task-queue/src/test/java/org/hawkular/metrics/tasks/impl/TaskSchedulerImplTest.java
    A task-queue/src/test/java/org/hawkular/metrics/tasks/impl/TaskSubscriber.java

  Log Message:
  -----------
  [HWKMETRICS-168] persist task to queue, update trigger api, and clean up tests

  Commit: 3af3148af82a2aa8227726fdbc9f8b7b42be5c74
      https://github.com/hawkular/hawkular-metrics/commit/3af3148af82a2aa8227726fdbc9f8b7b42be5c74
  Author: John Sanda <jsanda at redhat.com>
  Date:   2015-08-06 (Thu, 06 Aug 2015)

  Changed paths:
    M schema-manager/src/main/resources/schema.cql
    M task-queue/src/main/java/org/hawkular/metrics/tasks/impl/Lease.java
    M task-queue/src/main/java/org/hawkular/metrics/tasks/impl/LeaseService.java
    M task-queue/src/main/java/org/hawkular/metrics/tasks/impl/Queries.java
    M task-queue/src/main/java/org/hawkular/metrics/tasks/impl/TaskSchedulerImpl.java
    M task-queue/src/main/java/org/hawkular/metrics/tasks/impl/TaskServiceImpl.java
    M task-queue/src/test/java/org/hawkular/metrics/tasks/BaseITest.java
    M task-queue/src/test/java/org/hawkular/metrics/tasks/impl/LeaseServiceITest.java
    M task-queue/src/test/java/org/hawkular/metrics/tasks/impl/TaskSchedulerImplTest.java
    M task-queue/src/test/java/org/hawkular/metrics/tasks/impl/TaskServiceITest.java

  Log Message:
  -----------
  [HWKMETRICS-168] add method/test for scheduling task and update schema

Pretty much all of the code prior to this ticket is commented out because it is
basically being rewritten. Tests will be added back though.

  Commit: e39e33750590fd45a2271878660c4f362ba76caa
      https://github.com/hawkular/hawkular-metrics/commit/e39e33750590fd45a2271878660c4f362ba76caa
  Author: John Sanda <jsanda at redhat.com>
  Date:   2015-08-06 (Thu, 06 Aug 2015)

  Changed paths:
    M pom.xml
    M schema-manager/src/main/resources/schema.cql
    M task-queue/src/main/java/org/hawkular/metrics/tasks/api/RepeatingTrigger.java
    M task-queue/src/main/java/org/hawkular/metrics/tasks/api/TaskScheduler.java
    M task-queue/src/main/java/org/hawkular/metrics/tasks/api/Trigger.java
    M task-queue/src/main/java/org/hawkular/metrics/tasks/impl/Lease.java
    M task-queue/src/main/java/org/hawkular/metrics/tasks/impl/Queries.java
    M task-queue/src/main/java/org/hawkular/metrics/tasks/impl/Task2Impl.java
    M task-queue/src/main/java/org/hawkular/metrics/tasks/impl/TaskSchedulerImpl.java
    M task-queue/src/test/java/org/hawkular/metrics/tasks/BaseITest.java
    M task-queue/src/test/java/org/hawkular/metrics/tasks/impl/TaskSchedulerImplTest.java
    M task-queue/src/test/java/org/hawkular/metrics/tasks/impl/TaskSubscriber.java
    M task-queue/src/test/resources/log4j.xml

  Log Message:
  -----------
  [HWKMETRICS-168] big refactoring to get concurrency right.

The scheduler deals with 4 different thread pools - one for emitting ticks, one
for processing leases, one for executing tasks, and the C* driver's I/O
threads. Making sure things execute on the right thread pool turned out to be
a challenge when chaning various Rx operators together. It is not obvious like
when you explicitly submit some task to a thread pool.

  Commit: dcce78bc05b064b2e047d6dcbf2f201d3f44441e
      https://github.com/hawkular/hawkular-metrics/commit/dcce78bc05b064b2e047d6dcbf2f201d3f44441e
  Author: John Sanda <jsanda at redhat.com>
  Date:   2015-08-06 (Thu, 06 Aug 2015)

  Changed paths:
    M task-queue/src/main/java/org/hawkular/metrics/tasks/impl/TaskSchedulerImpl.java
    M task-queue/src/test/java/org/hawkular/metrics/tasks/impl/TaskSchedulerImplTest.java
    M task-queue/src/test/java/org/hawkular/metrics/tasks/impl/TaskSubscriber.java

  Log Message:
  -----------
  [HWKMETRICS-168] handle exceptions thrown by tasks

  Commit: bdec66d3b127baf8a2ba545ba9ae136dd567a61c
      https://github.com/hawkular/hawkular-metrics/commit/bdec66d3b127baf8a2ba545ba9ae136dd567a61c
  Author: John Sanda <jsanda at redhat.com>
  Date:   2015-08-06 (Thu, 06 Aug 2015)

  Changed paths:
    M schema-manager/src/main/resources/schema.cql
    M task-queue/src/main/java/org/hawkular/metrics/tasks/api/RepeatingTrigger.java
    M task-queue/src/main/java/org/hawkular/metrics/tasks/impl/TaskSchedulerImpl.java
    M task-queue/src/test/java/org/hawkular/metrics/tasks/impl/TaskSchedulerImplTest.java

  Log Message:
  -----------
  [HWKMETRICS-168] add support for setting number of executions in repeating trigger

This is generally useful feature, but I added it right now to help with
integration tests.

  Commit: 24517e223a2e1968c3dcbeae4034d34355922835
      https://github.com/hawkular/hawkular-metrics/commit/24517e223a2e1968c3dcbeae4034d34355922835
  Author: John Sanda <jsanda at redhat.com>
  Date:   2015-08-06 (Thu, 06 Aug 2015)

  Changed paths:
    M schema-manager/src/main/resources/schema.cql
    A task-queue/src/main/java/org/hawkular/metrics/tasks/api/SingleExecutionTrigger.java
    M task-queue/src/main/java/org/hawkular/metrics/tasks/api/Task2.java
    M task-queue/src/main/java/org/hawkular/metrics/tasks/impl/Queries.java
    M task-queue/src/main/java/org/hawkular/metrics/tasks/impl/Task2Impl.java
    M task-queue/src/main/java/org/hawkular/metrics/tasks/impl/TaskSchedulerImpl.java
    M task-queue/src/test/java/org/hawkular/metrics/tasks/impl/TaskSchedulerITest.java
    M task-queue/src/test/java/org/hawkular/metrics/tasks/impl/TaskSchedulerImplTest.java
    M task-queue/src/test/java/org/hawkular/metrics/tasks/impl/TaskSubscriber.java

  Log Message:
  -----------
  [HWKMETRICS-168] adding group_key and exec_order columns to task_queue table

These columns help if/when tasks have interdependencies. All tasks having the
same group key will be stored in the same queue, which means that they will be
associated with the same lease. The exec_order column defines an execution
order for tasks within the same group. Tasks with a lower number are executed
first.

The commit also adds/updates TaskSchedulerTest to use RxJava's TestScheduler.
It took me a good bit of time over the weekend to understand how to set things
up, but it was well worth the effort. Tests will be much more reliable,
consistent, and faster as they use a virtual clock.

  Commit: 20929c71edaea5f6636b48f345063d5e808c48f2
      https://github.com/hawkular/hawkular-metrics/commit/20929c71edaea5f6636b48f345063d5e808c48f2
  Author: John Sanda <jsanda at redhat.com>
  Date:   2015-08-06 (Thu, 06 Aug 2015)

  Changed paths:
    M task-queue/src/main/java/org/hawkular/metrics/tasks/api/TaskScheduler.java
    M task-queue/src/main/java/org/hawkular/metrics/tasks/impl/TaskSchedulerImpl.java
    M task-queue/src/test/java/org/hawkular/metrics/tasks/impl/TaskSchedulerITest.java

  Log Message:
  -----------
  [HWKMETRICS-168] emit event when scheduler is done with time slice

The getAvailableLeases method now publishes the timestamp when all work is
done. This is basically a test hook so that we can make tests consistent and
repeatable.

  Commit: 4b38e1aee164dd3568d3d1c92baed9d91d62c0c2
      https://github.com/hawkular/hawkular-metrics/commit/4b38e1aee164dd3568d3d1c92baed9d91d62c0c2
  Author: John Sanda <jsanda at redhat.com>
  Date:   2015-08-06 (Thu, 06 Aug 2015)

  Changed paths:
    M task-queue/src/main/java/org/hawkular/metrics/tasks/api/RepeatingTrigger.java
    M task-queue/src/main/java/org/hawkular/metrics/tasks/impl/TaskSchedulerImpl.java
    M task-queue/src/test/java/org/hawkular/metrics/tasks/impl/TaskSchedulerITest.java

  Log Message:
  -----------
  [HWKMETRICS-168] more tests

  Commit: df82328086bcdfc5221b2f613dbfbd9b98450671
      https://github.com/hawkular/hawkular-metrics/commit/df82328086bcdfc5221b2f613dbfbd9b98450671
  Author: John Sanda <jsanda at redhat.com>
  Date:   2015-08-06 (Thu, 06 Aug 2015)

  Changed paths:
    M api/metrics-api-jaxrs/src/main/java/org/hawkular/metrics/api/jaxrs/MetricsServiceLifecycle.java
    M core/metrics-core-impl/src/main/java/org/hawkular/metrics/core/impl/DataAccessImpl.java
    M core/metrics-core-impl/src/main/java/org/hawkular/metrics/core/impl/GenerateRate.java
    M core/metrics-core-impl/src/main/java/org/hawkular/metrics/core/impl/MetricsServiceImpl.java
    A core/metrics-core-impl/src/test/java/org/hawkular/metrics/core/impl/FakeTaskScheduler.java
    R core/metrics-core-impl/src/test/java/org/hawkular/metrics/core/impl/FakeTaskService.java
    M core/metrics-core-impl/src/test/java/org/hawkular/metrics/core/impl/MetricsServiceITest.java
    M core/metrics-core-impl/src/test/java/org/hawkular/metrics/core/impl/RatesITest.java
    M core/metrics-core-impl/src/test/resources/log4j.xml
    M task-queue/src/main/java/org/hawkular/metrics/tasks/api/RepeatingTrigger.java
    M task-queue/src/main/java/org/hawkular/metrics/tasks/api/SingleExecutionTrigger.java
    M task-queue/src/main/java/org/hawkular/metrics/tasks/api/TaskScheduler.java
    M task-queue/src/main/java/org/hawkular/metrics/tasks/impl/TaskSchedulerImpl.java
    M task-queue/src/test/java/org/hawkular/metrics/tasks/impl/TaskSchedulerITest.java
    R task-queue/src/test/java/org/hawkular/metrics/tasks/impl/TaskSchedulerImplTest.java

  Log Message:
  -----------
  [HWKMETRICS-168] get RatesITest passing again

  Commit: e9fafdb521a5c9a37844f027dc30e81cc24aa015
      https://github.com/hawkular/hawkular-metrics/commit/e9fafdb521a5c9a37844f027dc30e81cc24aa015
  Author: John Sanda <jsanda at redhat.com>
  Date:   2015-08-06 (Thu, 06 Aug 2015)

  Changed paths:
    M api/metrics-api-jaxrs/src/main/java/org/hawkular/metrics/api/jaxrs/MetricsServiceLifecycle.java
    M core/metrics-core-impl/src/main/java/org/hawkular/metrics/core/impl/MetricsServiceImpl.java
    R core/metrics-core-impl/src/main/java/org/hawkular/metrics/core/impl/TaskTypes.java
    M core/metrics-core-impl/src/test/java/org/hawkular/metrics/core/impl/RatesITest.java
    R task-queue/src/main/java/org/hawkular/metrics/tasks/api/Task.java
    R task-queue/src/main/java/org/hawkular/metrics/tasks/api/TaskExecutionException.java
    R task-queue/src/main/java/org/hawkular/metrics/tasks/api/TaskService.java
    R task-queue/src/main/java/org/hawkular/metrics/tasks/api/TaskServiceBuilder.java
    R task-queue/src/main/java/org/hawkular/metrics/tasks/api/TaskType.java
    R task-queue/src/main/java/org/hawkular/metrics/tasks/impl/TaskContainer.java
    R task-queue/src/main/java/org/hawkular/metrics/tasks/impl/TaskImpl.java
    R task-queue/src/main/java/org/hawkular/metrics/tasks/impl/TaskServiceImpl.java
    R task-queue/src/test/java/org/hawkular/metrics/tasks/impl/LeaseServiceITest.java
    R task-queue/src/test/java/org/hawkular/metrics/tasks/impl/TaskServiceITest.java

  Log Message:
  -----------
  [HWKMETRICS-168] removing obsolete classes

  Commit: b878edbb4dffac615f5da710efd0683ca23ca010
      https://github.com/hawkular/hawkular-metrics/commit/b878edbb4dffac615f5da710efd0683ca23ca010
  Author: John Sanda <jsanda at redhat.com>
  Date:   2015-08-06 (Thu, 06 Aug 2015)

  Changed paths:
    M api/metrics-api-jaxrs/src/main/java/org/hawkular/metrics/api/jaxrs/MetricsServiceLifecycle.java
    M core/metrics-core-impl/src/main/java/org/hawkular/metrics/core/impl/DataAccessImpl.java
    M core/metrics-core-impl/src/main/java/org/hawkular/metrics/core/impl/GenerateRate.java
    M core/metrics-core-impl/src/test/java/org/hawkular/metrics/core/impl/RatesITest.java
    M rest-tests/src/test/groovy/org/hawkular/metrics/rest/CountersITest.groovy
    M task-queue/src/test/java/org/hawkular/metrics/tasks/impl/TaskSchedulerITest.java

  Log Message:
  -----------
  [HWKMETRICS-168] fix post merge errors and disable integration test

There were some changes need as a result of HWKMETRICS-114 which made the
tenant id part of MetricId. The REST integration test for fetching counters is
disabled for now. Rates are calculated every minute (as of right now). We
do not want to have a test literally spin for several minutes waiting for rates
to be generated. We need a way to use RxJava's TestScheduler for REST
integration tests. I am going to createa spearate ticket for this because I
think it will involve a bit of work.

  Commit: a9ac133511cfd30576452a717b8f0f9fff39ea0b
      https://github.com/hawkular/hawkular-metrics/commit/a9ac133511cfd30576452a717b8f0f9fff39ea0b
  Author: John Sanda <jsanda at redhat.com>
  Date:   2015-08-06 (Thu, 06 Aug 2015)

  Changed paths:
    M core/metrics-core-impl/src/test/java/org/hawkular/metrics/core/impl/RatesITest.java
    M pom.xml
    A task-queue/src/main/java/org/hawkular/metrics/tasks/api/AbstractTrigger.java
    M task-queue/src/main/java/org/hawkular/metrics/tasks/api/RepeatingTrigger.java
    M task-queue/src/main/java/org/hawkular/metrics/tasks/api/SingleExecutionTrigger.java
    A task-queue/src/main/java/org/hawkular/metrics/tasks/api/TriggerBuilder.java
    M task-queue/src/test/java/org/hawkular/metrics/tasks/impl/TaskSchedulerITest.java

  Log Message:
  -----------
  [HWKMETRICS-168] remove duplicate code in trigger classes and bump RxJava version

  Commit: df08e49e4b3da59037f3ac4b0de3f4d622ca24a7
      https://github.com/hawkular/hawkular-metrics/commit/df08e49e4b3da59037f3ac4b0de3f4d622ca24a7
  Author: John Sanda <jsanda at redhat.com>
  Date:   2015-08-06 (Thu, 06 Aug 2015)

  Changed paths:
    M task-queue/src/main/java/org/hawkular/metrics/tasks/impl/TaskSchedulerImpl.java
    M task-queue/src/test/java/org/hawkular/metrics/tasks/impl/TaskSchedulerITest.java

  Log Message:
  -----------
  [HWKMETRICS-168] add test to verify execution order and fix query

I am adding a test that simulates a long running task. Every task is supposed
execute every minute. One of the tasks takes 3 minutes to complete. Tasks are
split into two groups. The test verifies that tests are executed in the
expected order despite the long delay.

This commit also fixes a bug in the query to find available leases. The filter
function needs to also check that the owern is not set in addition to the lease
not being finished.

  Commit: 6df62347f53b86ec30181c70942d4cb40e3d0555
      https://github.com/hawkular/hawkular-metrics/commit/6df62347f53b86ec30181c70942d4cb40e3d0555
  Author: John Sanda <jsanda at redhat.com>
  Date:   2015-08-06 (Thu, 06 Aug 2015)

  Changed paths:
    M core/metrics-core-impl/src/main/java/org/hawkular/metrics/core/impl/MetricsServiceImpl.java

  Log Message:
  -----------
  fix imports after rebase

Compare: https://github.com/hawkular/hawkular-metrics/compare/4467d6fb0b55...6df62347f53b