[Hawkular-commits] [hawkular/hawkular-metrics] 693f91: [HWKMETRICS-360] initial commit for new job-schedu...

Branch: refs/heads/master Home: https://github.com/hawkular/hawkular-metrics Commit: 693f919648b1d9aea996b2be63f32b3a4760d791 https://github.com/hawkular/hawkular-metrics/commit/693f919648b1d9aea996b... Author: John Sanda <jsanda(a)redhat.com&gt; Date: 2016-07-27 (Wed, 27 Jul 2016) Changed paths: M api/metrics-api-jaxrs/src/main/java/org/hawkular/metrics/api/jaxrs/MetricsServiceLifecycle.java M core/configuration-service/src/main/java/org/hawkular/metrics/sysconfig/ConfigurationService.java A core/datetime-service/pom.xml A core/datetime-service/src/main/java/org/hawkular/metrics/datetime/DateTimeService.java M core/metrics-core-service/pom.xml A core/metrics-core-service/src/main/java/org/hawkular/metrics/core/jobs/DeleteMetric.java A core/metrics-core-service/src/main/java/org/hawkular/metrics/core/jobs/JobsService.java A core/metrics-core-service/src/main/java/org/hawkular/metrics/core/jobs/JobsServiceImpl.java R core/metrics-core-service/src/main/java/org/hawkular/metrics/core/service/DateTimeService.java M core/metrics-core-service/src/main/java/org/hawkular/metrics/core/service/MetricsServiceImpl.java M core/metrics-core-service/src/test/java/org/hawkular/metrics/core/service/metrics/BaseMetricsITest.java R core/metrics-core-service/src/test/java/org/hawkular/metrics/core/service/metrics/GenerateRateITest.java R core/metrics-core-service/src/test/java/org/hawkular/metrics/core/service/metrics/RatesITest.java M core/metrics-core-service/src/test/resources/logback-test.xml M core/schema/src/main/java/org/hawkular/metrics/schema/SchemaService.java M core/schema/src/main/resources/org/hawkular/schema/cassalog.groovy M core/schema/src/main/resources/org/hawkular/schema/updates/schema-0.15.0.groovy A core/schema/src/main/resources/org/hawkular/schema/updates/schema-0.18.0.groovy R core/task-queue/pom.xml R core/task-queue/src/main/java/org/hawkular/metrics/tasks/DateTimeService.java R core/task-queue/src/main/java/org/hawkular/metrics/tasks/api/AbstractTrigger.java R core/task-queue/src/main/java/org/hawkular/metrics/tasks/api/RepeatingTrigger.java R core/task-queue/src/main/java/org/hawkular/metrics/tasks/api/SingleExecutionTrigger.java R core/task-queue/src/main/java/org/hawkular/metrics/tasks/api/Task2.java R core/task-queue/src/main/java/org/hawkular/metrics/tasks/api/TaskScheduler.java R core/task-queue/src/main/java/org/hawkular/metrics/tasks/api/Trigger.java R core/task-queue/src/main/java/org/hawkular/metrics/tasks/api/TriggerBuilder.java R core/task-queue/src/main/java/org/hawkular/metrics/tasks/impl/Lease.java R core/task-queue/src/main/java/org/hawkular/metrics/tasks/impl/LeaseService.java R core/task-queue/src/main/java/org/hawkular/metrics/tasks/impl/Queries.java R core/task-queue/src/main/java/org/hawkular/metrics/tasks/impl/Task2Impl.java R core/task-queue/src/main/java/org/hawkular/metrics/tasks/impl/TaskSchedulerImpl.java R core/task-queue/src/main/java/org/hawkular/metrics/tasks/log/TaskQueueLogger.java R core/task-queue/src/main/java/org/hawkular/metrics/tasks/log/TaskQueueLogging.java R core/task-queue/src/test/java/org/hawkular/metrics/tasks/BaseITest.java R core/task-queue/src/test/java/org/hawkular/metrics/tasks/impl/TaskSchedulerITest.java R core/task-queue/src/test/java/org/hawkular/metrics/tasks/impl/TaskSubscriber.java R core/task-queue/src/test/resources/logback-test.xml M integration-tests/rest-tests-jaxrs/pom.xml M integration-tests/rest-tests-jaxrs/src/test/groovy/org/hawkular/metrics/rest/AvailabilityMetricStatisticsITest.groovy M integration-tests/rest-tests-jaxrs/src/test/groovy/org/hawkular/metrics/rest/CassandraBackendITest.groovy M integration-tests/rest-tests-jaxrs/src/test/groovy/org/hawkular/metrics/rest/CountersITest.groovy M integration-tests/rest-tests-jaxrs/src/test/groovy/org/hawkular/metrics/rest/GaugeMetricStatisticsITest.groovy M integration-tests/rest-tests-jaxrs/src/test/groovy/org/hawkular/metrics/rest/TenantITest.groovy A job-scheduler/pom.xml A job-scheduler/src/main/java/org/hawkular/metrics/scheduler/api/JobDetails.java A job-scheduler/src/main/java/org/hawkular/metrics/scheduler/api/RepeatingTrigger.java A job-scheduler/src/main/java/org/hawkular/metrics/scheduler/api/Scheduler.java A job-scheduler/src/main/java/org/hawkular/metrics/scheduler/api/SingleExecutionTrigger.java A job-scheduler/src/main/java/org/hawkular/metrics/scheduler/api/Trigger.java A job-scheduler/src/main/java/org/hawkular/metrics/scheduler/impl/LockManager.java A job-scheduler/src/main/java/org/hawkular/metrics/scheduler/impl/SchedulerImpl.java A job-scheduler/src/test/java/org/hawkular/metrics/scheduler/impl/JobExecutionTest.java A job-scheduler/src/test/java/org/hawkular/metrics/scheduler/impl/JobSchedulerTest.java A job-scheduler/src/test/java/org/hawkular/metrics/scheduler/impl/JobSchedulingTest.java A job-scheduler/src/test/resources/logback-test.xml A job-scheduler/src/test/resources/testng.xml M pom.xml Log Message: ----------- [HWKMETRICS-360] initial commit for new job-scheduler module This commit includes some initial functionality for scheduling jobs. I have also pulled DateTimeService out into a separate module so that it can be used by both metrics-core-service and by job-scheduler. [HWKMETRICS-360] fix merge conflicts from rebase and use config service I have also updated ConfigurationService to use LOCAL_QUORUM consistency and added a method for updating a single property. [HWKMETRICS-360] initial support for executing single execution jobs This commit adds support for executing single execution or non-repeating jobs. The job scheduler looks for and executes jobs by minute time slices. Every minute a new task is kicked off to find jobs to execute. The active, i.e., oldest time slice is stored in the sys_config table. The task will first check for jobs in that time slice. If there are available jobs, the scheduler will obtain the necessary locks and execute the jobs. If there are no jobs or if all of the jobs are already running the scheduler will look for jobs in following time slices. This is still pretty rough around the edges. [HWKMETRICS-360] initial support for executing repeating jobs There are a few important changes for handling repeating jobs. The first and probably most obvious is having to reschedule the job. This involves updating the jobs and the jobs_status tables. Next, when job execution is done, we need to release the job lock. We use thread names for lock values. Since we don't know which thread is running when a lock is acquired, we need to remember the lock value in some fashion. The current solution is not so great as it basically involves a global variable. Lastly, we need to remove the job id from the activeJobs set created in the start method. If we do not do this, we could block subsequent executions of the job. Some minor changes were made to LockManager. There are methods acquiring locks and for acquiring permits. Permits are used for shared or multi access as in the case of scheduling jobs. We want to allow multiple clients schedule jobs concurrently. For executing a job, we want an exclusive lock that has a single owner. Right now I am using the same table for both of these. We'll see how it goes. [HWKMETRICS-360] try to make tests more robust Previously tests were using a shared job scheduler instance. I started running into problems once I added tests for repeating jobs. Jobs set up from one test method were running in later tests and causing problems. I decided that a new job scheduler instance with a clean db per test would be better. [HWKMETRICS-360] finally work around deadlock issues I still do not understand but in my unit tests I was hitting deadlock issues. I had to move the execution of most queries off of the computation scheduler and onto a different scheduler. [HWKMETRICS-360] make the job factories return a Completable Completable is a better type to use for the job execution code. It is a more natural fit for async execution of code that involves side effects. committing local edits for others to see [HKKMETRICS-360] some minor clean up, remove unneeded code [HWKMETRICS-360] fix checkstyle, license errors [HWKMETRICS-360] fix version numbers [HWKMETRICS-360] turn on debug logging for travis, configure RejectedExecutionHandler [HWKMETRICS-360] initialze TestScheduler for each test [HWKMETRICS-360] fix timing issue in executeLongRunningSingleExecutionJob() [HWKMETRICS-360] fix missing dependency after rebase [HWKMETRICS-360] add missing import [HWKMETRICS-360] more refactoring/debugging [HWKMETRICS-360] get post job execution clean up working with Completable more debugging increase timeout to avoid race condition clean up obsolete code and add some comments This is squash commit of several small changes in an effort to eliminate intermittent test failures. fix version for datetime service remove duplicate dependency declaration serialize access to activeTimeSlices cache remove unused debug code that was accidentally committed remove legacy, unused task scheduler code remove configuration service since it is no longer used in scheduler initial commit for JobsService JobsService lives in metrics-core-service. It serves a couple purposes. It encapsulates the scheduler and controls its lifecycle. JobsService will also provide APIs for submitting and canceling jobs. fix license headers [HWKMETRICS-360] add some debug logging and use concat instead of merge SchedulerImpl.doJobExecution() was previously using Completable.merge which allows for out of order execution of the Completable arguments. In testing I discovered that notifications of job completion were being sent before job execution completed. This was due to using merge. Completable.concat fixes this. a little clean up, trying to get github to see my previous commit [HWKMETRICS-360] use Observable version of updateActiveTimeSlices to avoid eager execution Commit: 703e616735c2f262fcf834d8c34555c54dd5421a https://github.com/hawkular/hawkular-metrics/commit/703e616735c2f262fcf83... Author: Michael Burman <yak(a)iki.fi&gt; Date: 2016-07-27 (Wed, 27 Jul 2016) Changed paths: M api/metrics-api-jaxrs/src/main/java/org/hawkular/metrics/api/jaxrs/MetricsServiceLifecycle.java M core/configuration-service/src/main/java/org/hawkular/metrics/sysconfig/ConfigurationService.java A core/datetime-service/pom.xml A core/datetime-service/src/main/java/org/hawkular/metrics/datetime/DateTimeService.java M core/metrics-core-service/pom.xml A core/metrics-core-service/src/main/java/org/hawkular/metrics/core/jobs/DeleteMetric.java A core/metrics-core-service/src/main/java/org/hawkular/metrics/core/jobs/JobsService.java A core/metrics-core-service/src/main/java/org/hawkular/metrics/core/jobs/JobsServiceImpl.java R core/metrics-core-service/src/main/java/org/hawkular/metrics/core/service/DateTimeService.java M core/metrics-core-service/src/main/java/org/hawkular/metrics/core/service/MetricsServiceImpl.java M core/metrics-core-service/src/test/java/org/hawkular/metrics/core/service/metrics/BaseMetricsITest.java R core/metrics-core-service/src/test/java/org/hawkular/metrics/core/service/metrics/GenerateRateITest.java R core/metrics-core-service/src/test/java/org/hawkular/metrics/core/service/metrics/RatesITest.java M core/metrics-core-service/src/test/resources/logback-test.xml M core/schema/src/main/java/org/hawkular/metrics/schema/SchemaService.java M core/schema/src/main/resources/org/hawkular/schema/cassalog.groovy M core/schema/src/main/resources/org/hawkular/schema/updates/schema-0.15.0.groovy A core/schema/src/main/resources/org/hawkular/schema/updates/schema-0.18.0.groovy R core/task-queue/pom.xml R core/task-queue/src/main/java/org/hawkular/metrics/tasks/DateTimeService.java R core/task-queue/src/main/java/org/hawkular/metrics/tasks/api/AbstractTrigger.java R core/task-queue/src/main/java/org/hawkular/metrics/tasks/api/RepeatingTrigger.java R core/task-queue/src/main/java/org/hawkular/metrics/tasks/api/SingleExecutionTrigger.java R core/task-queue/src/main/java/org/hawkular/metrics/tasks/api/Task2.java R core/task-queue/src/main/java/org/hawkular/metrics/tasks/api/TaskScheduler.java R core/task-queue/src/main/java/org/hawkular/metrics/tasks/api/Trigger.java R core/task-queue/src/main/java/org/hawkular/metrics/tasks/api/TriggerBuilder.java R core/task-queue/src/main/java/org/hawkular/metrics/tasks/impl/Lease.java R core/task-queue/src/main/java/org/hawkular/metrics/tasks/impl/LeaseService.java R core/task-queue/src/main/java/org/hawkular/metrics/tasks/impl/Queries.java R core/task-queue/src/main/java/org/hawkular/metrics/tasks/impl/Task2Impl.java R core/task-queue/src/main/java/org/hawkular/metrics/tasks/impl/TaskSchedulerImpl.java R core/task-queue/src/main/java/org/hawkular/metrics/tasks/log/TaskQueueLogger.java R core/task-queue/src/main/java/org/hawkular/metrics/tasks/log/TaskQueueLogging.java R core/task-queue/src/test/java/org/hawkular/metrics/tasks/BaseITest.java R core/task-queue/src/test/java/org/hawkular/metrics/tasks/impl/TaskSchedulerITest.java R core/task-queue/src/test/java/org/hawkular/metrics/tasks/impl/TaskSubscriber.java R core/task-queue/src/test/resources/logback-test.xml M integration-tests/rest-tests-jaxrs/pom.xml M integration-tests/rest-tests-jaxrs/src/test/groovy/org/hawkular/metrics/rest/AvailabilityMetricStatisticsITest.groovy M integration-tests/rest-tests-jaxrs/src/test/groovy/org/hawkular/metrics/rest/CassandraBackendITest.groovy M integration-tests/rest-tests-jaxrs/src/test/groovy/org/hawkular/metrics/rest/CountersITest.groovy M integration-tests/rest-tests-jaxrs/src/test/groovy/org/hawkular/metrics/rest/GaugeMetricStatisticsITest.groovy M integration-tests/rest-tests-jaxrs/src/test/groovy/org/hawkular/metrics/rest/TenantITest.groovy A job-scheduler/pom.xml A job-scheduler/src/main/java/org/hawkular/metrics/scheduler/api/JobDetails.java A job-scheduler/src/main/java/org/hawkular/metrics/scheduler/api/RepeatingTrigger.java A job-scheduler/src/main/java/org/hawkular/metrics/scheduler/api/Scheduler.java A job-scheduler/src/main/java/org/hawkular/metrics/scheduler/api/SingleExecutionTrigger.java A job-scheduler/src/main/java/org/hawkular/metrics/scheduler/api/Trigger.java A job-scheduler/src/main/java/org/hawkular/metrics/scheduler/impl/LockManager.java A job-scheduler/src/main/java/org/hawkular/metrics/scheduler/impl/SchedulerImpl.java A job-scheduler/src/test/java/org/hawkular/metrics/scheduler/impl/JobExecutionTest.java A job-scheduler/src/test/java/org/hawkular/metrics/scheduler/impl/JobSchedulerTest.java A job-scheduler/src/test/java/org/hawkular/metrics/scheduler/impl/JobSchedulingTest.java A job-scheduler/src/test/resources/logback-test.xml A job-scheduler/src/test/resources/testng.xml M pom.xml Log Message: ----------- Merge pull request #549 from jsanda/hwkmetrics-360 [HWKMETRICS-360] initial job scheduler work Compare: https://github.com/hawkular/hawkular-metrics/compare/65404bcc24e7...703e6...