[Hawkular-commits] [hawkular/hawkular-metrics] bf3cce: [HWKMETRICS-168] initial commit for new scheduler ...

[John Sanda]

  Branch: refs/heads/task-scheduler
  Home:   https://github.com/hawkular/hawkular-metrics
  Commit: bf3cceb64c638ff719b06004a92e9cd2ebc070f9
      https://github.com/hawkular/hawkular-metrics/commit/bf3cceb64c638ff719b06004a92e9cd2ebc070f9
  Author: John Sanda <jsanda at redhat.com>
  Date:   2015-08-06 (Thu, 06 Aug 2015)

  Changed paths:
    M api/metrics-api-jaxrs/src/main/java/org/hawkular/metrics/api/jaxrs/MetricsServiceLifecycle.java
    M core/metrics-core-impl/src/main/java/org/hawkular/metrics/core/impl/GenerateRate.java
    M core/metrics-core-impl/src/main/java/org/hawkular/metrics/core/impl/MetricsServiceImpl.java
    R core/metrics-core-impl/src/main/java/org/hawkular/metrics/core/impl/TaskTypes.java
    A core/metrics-core-impl/src/test/java/org/hawkular/metrics/core/impl/FakeTaskScheduler.java
    R core/metrics-core-impl/src/test/java/org/hawkular/metrics/core/impl/FakeTaskService.java
    M core/metrics-core-impl/src/test/java/org/hawkular/metrics/core/impl/MetricsServiceITest.java
    M core/metrics-core-impl/src/test/java/org/hawkular/metrics/core/impl/RatesITest.java
    M core/metrics-core-impl/src/test/resources/log4j.xml
    M pom.xml
    M rest-tests/src/test/groovy/org/hawkular/metrics/rest/CountersITest.groovy
    M schema-manager/src/main/resources/schema.cql
    A task-queue/src/main/java/org/hawkular/metrics/tasks/api/AbstractTrigger.java
    A task-queue/src/main/java/org/hawkular/metrics/tasks/api/RepeatingTrigger.java
    A task-queue/src/main/java/org/hawkular/metrics/tasks/api/SingleExecutionTrigger.java
    R task-queue/src/main/java/org/hawkular/metrics/tasks/api/Task.java
    A task-queue/src/main/java/org/hawkular/metrics/tasks/api/Task2.java
    R task-queue/src/main/java/org/hawkular/metrics/tasks/api/TaskExecutionException.java
    A task-queue/src/main/java/org/hawkular/metrics/tasks/api/TaskScheduler.java
    R task-queue/src/main/java/org/hawkular/metrics/tasks/api/TaskService.java
    R task-queue/src/main/java/org/hawkular/metrics/tasks/api/TaskServiceBuilder.java
    R task-queue/src/main/java/org/hawkular/metrics/tasks/api/TaskType.java
    A task-queue/src/main/java/org/hawkular/metrics/tasks/api/Trigger.java
    A task-queue/src/main/java/org/hawkular/metrics/tasks/api/TriggerBuilder.java
    M task-queue/src/main/java/org/hawkular/metrics/tasks/impl/Lease.java
    M task-queue/src/main/java/org/hawkular/metrics/tasks/impl/LeaseService.java
    M task-queue/src/main/java/org/hawkular/metrics/tasks/impl/Queries.java
    A task-queue/src/main/java/org/hawkular/metrics/tasks/impl/Task2Impl.java
    R task-queue/src/main/java/org/hawkular/metrics/tasks/impl/TaskContainer.java
    R task-queue/src/main/java/org/hawkular/metrics/tasks/impl/TaskImpl.java
    A task-queue/src/main/java/org/hawkular/metrics/tasks/impl/TaskSchedulerImpl.java
    R task-queue/src/main/java/org/hawkular/metrics/tasks/impl/TaskServiceImpl.java
    M task-queue/src/test/java/org/hawkular/metrics/tasks/BaseITest.java
    R task-queue/src/test/java/org/hawkular/metrics/tasks/impl/LeaseServiceITest.java
    M task-queue/src/test/java/org/hawkular/metrics/tasks/impl/TaskSchedulerITest.java
    R task-queue/src/test/java/org/hawkular/metrics/tasks/impl/TaskServiceITest.java
    A task-queue/src/test/java/org/hawkular/metrics/tasks/impl/TaskSubscriber.java
    M task-queue/src/test/resources/log4j.xml

  Log Message:
  -----------
  [HWKMETRICS-168] initial commit for new scheduler and task classes

[HWKMETRICS-168] persist task to queue, update trigger api, and clean up tests

[HWKMETRICS-168] add method/test for scheduling task and update schema

Pretty much all of the code prior to this ticket is commented out because it is
basically being rewritten. Tests will be added back though.

[HWKMETRICS-168] big refactoring to get concurrency right.

The scheduler deals with 4 different thread pools - one for emitting ticks, one
for processing leases, one for executing tasks, and the C* driver's I/O
threads. Making sure things execute on the right thread pool turned out to be
a challenge when chaning various Rx operators together. It is not obvious like
when you explicitly submit some task to a thread pool.

[HWKMETRICS-168] handle exceptions thrown by tasks

[HWKMETRICS-168] add support for setting number of executions in repeating trigger

This is generally useful feature, but I added it right now to help with
integration tests.

[HWKMETRICS-168] adding group_key and exec_order columns to task_queue table

These columns help if/when tasks have interdependencies. All tasks having the
same group key will be stored in the same queue, which means that they will be
associated with the same lease. The exec_order column defines an execution
order for tasks within the same group. Tasks with a lower number are executed
first.

The commit also adds/updates TaskSchedulerTest to use RxJava's TestScheduler.
It took me a good bit of time over the weekend to understand how to set things
up, but it was well worth the effort. Tests will be much more reliable,
consistent, and faster as they use a virtual clock.

[HWKMETRICS-168] emit event when scheduler is done with time slice

The getAvailableLeases method now publishes the timestamp when all work is
done. This is basically a test hook so that we can make tests consistent and
repeatable.

[HWKMETRICS-168] more tests

[HWKMETRICS-168] get RatesITest passing again

[HWKMETRICS-168] removing obsolete classes

[HWKMETRICS-168] fix post merge errors and disable integration test

There were some changes need as a result of HWKMETRICS-114 which made the
tenant id part of MetricId. The REST integration test for fetching counters is
disabled for now. Rates are calculated every minute (as of right now). We
do not want to have a test literally spin for several minutes waiting for rates
to be generated. We need a way to use RxJava's TestScheduler for REST
integration tests. I am going to createa spearate ticket for this because I
think it will involve a bit of work.

[HWKMETRICS-168] remove duplicate code in trigger classes and bump RxJava version

[HWKMETRICS-168] add test to verify execution order and fix query

I am adding a test that simulates a long running task. Every task is supposed
execute every minute. One of the tasks takes 3 minutes to complete. Tasks are
split into two groups. The test verifies that tests are executed in the
expected order despite the long delay.

This commit also fixes a bug in the query to find available leases. The filter
function needs to also check that the owern is not set in addition to the lease
not being finished.

fix imports after rebase