[jboss-jira] [JBoss JIRA] (JBRULES-3473) Ability to resume a benchmark

[Geoffrey De Smet (JIRA)]

    [ https://issues.jboss.org/browse/JBRULES-3473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12686331#comment-12686331 ] 

Geoffrey De Smet edited comment on JBRULES-3473 at 4/22/12 11:12 AM:
---------------------------------------------------------------------

Good idea. I often kill the Benchmarker process when I want all CPU resources available for my IDE, losing hours of benchmarking CPU time.


Some things to pounder about:
- Any solution for this issue and JBRULES-3477 will involve serializing benchmark results (including statistics) to disk.
- best solution changed statistics data is now kept in memory for every solver for every file. Only at the end the graphs and csv's are made. Is this asking for a OOM for some future use cases? (yes)
- Are the benchmarks being done in the correct order?
Currently: per solver, then per file.
If it's the other way around: per file, then per solver, we can output best solution changed statistics graphs sooner (and release that memory!)
- "A new API is added to Planner that allows to resume the benchmark from the above intermediate file." I doubt the usability of this exact way. We can make it work automatically, like "<resumeFromLastIfThatIsUnfinished>true</>" (over-ridable by system property):
-- Planner knows what the last benchmark in the benchmarkDirectory, because the benchmarks are in timestamped subdirs
-- Problem is what if the benchmark config has changed meanwhile? We need some way to detect if any of the benchmarks already run are the exactly the same. But what if the drools-core jar or planner jar or any of the classes have been updated? Does it count as being the same benchmark?
-- resumeFromLastIfThatIsUnfinished must be overridable by a system property, so you can just configure a run config that way without having to change the xml every time to enable/disable it.


                
      was (Author: ge0ffrey):
    Good idea. I often kill the Benchmarker process when I want all CPU resources available for my IDE, losing hours of benchmarking CPU time.


Some things to pounder about:
- Any solution for this issue and JBRULES-3477 will involve serializing benchmark results (including statistics) to disk.
- best solution changed statistics data is kept in memory for every solver for every file. Only at the end the graphs and csv's are made. Is this asking for a OOM?
- Are the benchmarks being done in the correct order?
Currently: per solver, then per file.
If it's the other way around: per file, then per solver, we can output best solution changed statistics graphs sooner (and release that memory!)
- "A new API is added to Planner that allows to resume the benchmark from the above intermediate file." I doubt the usability of this exact way. We can make it work automatically, like "<resumeFromLastIfThatIsUnfinished>true</>" (over-ridable by system property):
-- Planner knows what the last benchmark in the benchmarkDirectory, because the benchmarks are in timestamped subdirs
-- Problem is what if the benchmark config has changed meanwhile? We need some way to detect if any of the benchmarks already run are the exactly the same. But what if the drools-core jar or planner jar or any of the classes have been updated? Does it count as being the same benchmark?
-- resumeFromLastIfThatIsUnfinished must be overridable by a system property, so you can just configure a run config that way without having to change the xml every time to enable/disable it.


                  
> Ability to resume a benchmark
> -----------------------------
>
>                 Key: JBRULES-3473
>                 URL: https://issues.jboss.org/browse/JBRULES-3473
>             Project: Drools
>          Issue Type: Feature Request
>      Security Level: Public(Everyone can see) 
>          Components: drools-planner
>            Reporter: Lukáš Petrovický
>            Assignee: Geoffrey De Smet
>
> Benchmarks are usually long-running processes. During the runtime of a benchmark, lots of things can happen - you may lose power on your laptop, you may accidentally kill the benchmark process... Those have already happened to me and when they do, I lose the hours of benchmarks that have run so far but haven't yet generated a HTML report.
> What I propose is the following:
> - After each benchmark in a suite finishes, some kind of intermediate file is written, describing exactly which other benchmarks are left to do. It may also need to store the results of the already finished benchmarks, so that in the future we can create the charts etc.
> - A new API is added to Planner that allows to resume the benchmark from the above intermediate file.
> - Possibly also another API for "suspending" the benchmarking process, ie. persisting it into the intermediate file and halting.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira