[Hawkular-commits] [hawkular/hawkular-metrics] bf3cce: [HWKMETRICS-168] initial commit for new scheduler ...

Branch: refs/heads/task-scheduler Home: https://github.com/hawkular/hawkular-metrics Commit: bf3cceb64c638ff719b06004a92e9cd2ebc070f9 https://github.com/hawkular/hawkular-metrics/commit/bf3cceb64c638ff719b06... Author: John Sanda <jsanda(a)redhat.com&gt; Date: 2015-08-06 (Thu, 06 Aug 2015) Changed paths: M api/metrics-api-jaxrs/src/main/java/org/hawkular/metrics/api/jaxrs/MetricsServiceLifecycle.java M core/metrics-core-impl/src/main/java/org/hawkular/metrics/core/impl/GenerateRate.java M core/metrics-core-impl/src/main/java/org/hawkular/metrics/core/impl/MetricsServiceImpl.java R core/metrics-core-impl/src/main/java/org/hawkular/metrics/core/impl/TaskTypes.java A core/metrics-core-impl/src/test/java/org/hawkular/metrics/core/impl/FakeTaskScheduler.java R core/metrics-core-impl/src/test/java/org/hawkular/metrics/core/impl/FakeTaskService.java M core/metrics-core-impl/src/test/java/org/hawkular/metrics/core/impl/MetricsServiceITest.java M core/metrics-core-impl/src/test/java/org/hawkular/metrics/core/impl/RatesITest.java M core/metrics-core-impl/src/test/resources/log4j.xml M pom.xml M rest-tests/src/test/groovy/org/hawkular/metrics/rest/CountersITest.groovy M schema-manager/src/main/resources/schema.cql A task-queue/src/main/java/org/hawkular/metrics/tasks/api/AbstractTrigger.java A task-queue/src/main/java/org/hawkular/metrics/tasks/api/RepeatingTrigger.java A task-queue/src/main/java/org/hawkular/metrics/tasks/api/SingleExecutionTrigger.java R task-queue/src/main/java/org/hawkular/metrics/tasks/api/Task.java A task-queue/src/main/java/org/hawkular/metrics/tasks/api/Task2.java R task-queue/src/main/java/org/hawkular/metrics/tasks/api/TaskExecutionException.java A task-queue/src/main/java/org/hawkular/metrics/tasks/api/TaskScheduler.java R task-queue/src/main/java/org/hawkular/metrics/tasks/api/TaskService.java R task-queue/src/main/java/org/hawkular/metrics/tasks/api/TaskServiceBuilder.java R task-queue/src/main/java/org/hawkular/metrics/tasks/api/TaskType.java A task-queue/src/main/java/org/hawkular/metrics/tasks/api/Trigger.java A task-queue/src/main/java/org/hawkular/metrics/tasks/api/TriggerBuilder.java M task-queue/src/main/java/org/hawkular/metrics/tasks/impl/Lease.java M task-queue/src/main/java/org/hawkular/metrics/tasks/impl/LeaseService.java M task-queue/src/main/java/org/hawkular/metrics/tasks/impl/Queries.java A task-queue/src/main/java/org/hawkular/metrics/tasks/impl/Task2Impl.java R task-queue/src/main/java/org/hawkular/metrics/tasks/impl/TaskContainer.java R task-queue/src/main/java/org/hawkular/metrics/tasks/impl/TaskImpl.java A task-queue/src/main/java/org/hawkular/metrics/tasks/impl/TaskSchedulerImpl.java R task-queue/src/main/java/org/hawkular/metrics/tasks/impl/TaskServiceImpl.java M task-queue/src/test/java/org/hawkular/metrics/tasks/BaseITest.java R task-queue/src/test/java/org/hawkular/metrics/tasks/impl/LeaseServiceITest.java M task-queue/src/test/java/org/hawkular/metrics/tasks/impl/TaskSchedulerITest.java R task-queue/src/test/java/org/hawkular/metrics/tasks/impl/TaskServiceITest.java A task-queue/src/test/java/org/hawkular/metrics/tasks/impl/TaskSubscriber.java M task-queue/src/test/resources/log4j.xml Log Message: ----------- [HWKMETRICS-168] initial commit for new scheduler and task classes [HWKMETRICS-168] persist task to queue, update trigger api, and clean up tests [HWKMETRICS-168] add method/test for scheduling task and update schema Pretty much all of the code prior to this ticket is commented out because it is basically being rewritten. Tests will be added back though. [HWKMETRICS-168] big refactoring to get concurrency right. The scheduler deals with 4 different thread pools - one for emitting ticks, one for processing leases, one for executing tasks, and the C* driver's I/O threads. Making sure things execute on the right thread pool turned out to be a challenge when chaning various Rx operators together. It is not obvious like when you explicitly submit some task to a thread pool. [HWKMETRICS-168] handle exceptions thrown by tasks [HWKMETRICS-168] add support for setting number of executions in repeating trigger This is generally useful feature, but I added it right now to help with integration tests. [HWKMETRICS-168] adding group_key and exec_order columns to task_queue table These columns help if/when tasks have interdependencies. All tasks having the same group key will be stored in the same queue, which means that they will be associated with the same lease. The exec_order column defines an execution order for tasks within the same group. Tasks with a lower number are executed first. The commit also adds/updates TaskSchedulerTest to use RxJava's TestScheduler. It took me a good bit of time over the weekend to understand how to set things up, but it was well worth the effort. Tests will be much more reliable, consistent, and faster as they use a virtual clock. [HWKMETRICS-168] emit event when scheduler is done with time slice The getAvailableLeases method now publishes the timestamp when all work is done. This is basically a test hook so that we can make tests consistent and repeatable. [HWKMETRICS-168] more tests [HWKMETRICS-168] get RatesITest passing again [HWKMETRICS-168] removing obsolete classes [HWKMETRICS-168] fix post merge errors and disable integration test There were some changes need as a result of HWKMETRICS-114 which made the tenant id part of MetricId. The REST integration test for fetching counters is disabled for now. Rates are calculated every minute (as of right now). We do not want to have a test literally spin for several minutes waiting for rates to be generated. We need a way to use RxJava's TestScheduler for REST integration tests. I am going to createa spearate ticket for this because I think it will involve a bit of work. [HWKMETRICS-168] remove duplicate code in trigger classes and bump RxJava version [HWKMETRICS-168] add test to verify execution order and fix query I am adding a test that simulates a long running task. Every task is supposed execute every minute. One of the tasks takes 3 minutes to complete. Tasks are split into two groups. The test verifies that tests are executed in the expected order despite the long delay. This commit also fixes a bug in the query to find available leases. The filter function needs to also check that the owern is not set in addition to the lease not being finished. fix imports after rebase