I've revamped the tests in this way so far:

• implemented "download" multi-task (could be exposed as grunt module)
  • leaving aside its name, it does actually much more: 1. downloads distribution, 2. validates checksum, 3. extracts, 4. extracts configuration file (overlay), 5. renames to the stable name (e.g. runtimes/apache-activemq instead of apache-activemq-5.8.0)
• ability to kill the process running on background
  • each need to use different approach
    • activemq runs as daemon, is needs to be killed by running 'activemq stop' command
    • vertx runs as process, can be killed simply by issuing SIGTERM
  • when grunt ends prematurily (by killing its process or by test failure) the server is stopped gracefully (that requires grunt:0.4.1 as highter versions contains a bug/limitation)
  • we are able to stop the server after test by executing :stop subtask
• tests rewritten to Karma
  • that wasn't intentional first, but Karma has another advantage - it uses its own server and bypasses browser security (CORS) so testing is much easier
    • as connect can't be killed gracefully after the test (at least not simple way)
  • Karma has one downside though, it can't load HTML file with QUnit script (at least not vanilla Karma), because it expects files array in the config

I believe this approach has certain advantages:

• configuration for downloads is way simpler (declarative)
  • all you need is declaring http urls
  • it is also smart: installs new runtime just when needed
  • because it uses temporary folder for extraction we know exactly when the distribution is ready to be moved to its position (process-kill-proof solution)
• running tasks is faster and more stable (in theory, the time will tell us)
  • its faster because it waits for certain log in process' stdout (as oppose to waiting hard-coded delay time)
  • its more stable because thanks to using grunt-external-daemon the processes are handled correctly
• it can be simply debugged
  • grunt ... --verbose --debug
• the only what aerogear-js needs to do is make sure aerogear-js-integration is cloned and then run grunt ci:activemq ci:vertx

Some disadvantages:

• none
  • just kidding, here it is:
• using forked version of grunt-external-daemon
  • the original didn't handle kill processes and didn't support a way of stopping that activemq requires as a daemon process
  • but we should contribute this improvement back
  • the project doesn't use cleaneast project ever and I made it even bigger mess, some re-structure would be nice after agreement with an original author

The code is here:

aerogear-js: https://github.com/lfryc/aerogear-js/tree/AGJS-211-rework-integration-tests

aerogear-js-integration: https://github.com/lfryc/aerogear-js-integration/tree/AGJS-211-rework-integration-tests

Some polishing is still needed as it is still rather proof of concept.
Old configs and dependencies need to be removed.

Some commands to play with after clones:

• grunt ci:vertx ci:activemq
• grunt ci:vertx --verbose --debug