[wildfly-dev] Design Proposal: Build split and provisioning

This design proposal covers the inter related tasks of splitting up the build, and also creating a build/provisioning system that will make it easy for end users to consume Wildfly. Apologies for the length, but it is a complex topic. The first part explains what we are trying to achieve, the second part covers how we are planning to actually implement it. The Wildfly code base is over a million lines of java and has a test suite that generally takes close to two hours to run in its entirety. This makes the project very unwieldily, and the large size and slow test suite makes development painful. To deal with this issue we are going to split the Wildfly code base into smaller discrete repositories. The planned split is as follows: - Core: just the WF core - Arquillian: the arquillian adaptors - Servlet: a WF distribution with just Undertow, and some basic EE functionality such as naming - EE: All the core EE related functionality, EJB's, messaging etc - Clustering: The core clustering functionality - Console: The management console - Dist: brings all the pieces together, and allows us to run all tests against a full server Note that this list is in no way final, and is open to debate. We will most likely want to split up the EE component at some point, possibly along some kind of web profile/full profile type split. Each of these repos will build a feature pack, which will contain the following: - Feature specification / description - Core version requirements (e.g. WF10) - Dependency info on other features (e.g. RestEASY X requires CDI 1.1) - module.xml files for all required modules that are not provided by other features - References to maven GAV's for jars (possibly a level of indirection here, module.xml may just contain the group and artifact, and the version may be in a version.properties file to allow it to be easily overridden) - Default configuration snippet, subsystem snippets are packaged in the subsystem jars, templates that combine them into config files are part of the feature pack. - Misc files (e.g. xsds) with indication of where on path to place them Note that a feature pack is not a complete server, it cannot simply be extracted and run, it first needs to be assembled into a server by the provisioning tool. The feature packs also just contain references to the maven GAV of required jars, they do not have the actual jars in the pack (which should make them very lightweight). Feature packs will be assembled by the WF build tool, which is just a maven plugin that will replace our existing hacky collection of ant scripts. Actual server instances will be assembled by the provisioning tool, which will be implemented as a library with several different front ends, including a maven plugin and a CLI (possibly integrated into our existing CLI). In general the provisioning tool will be able to provision three different types of servers: - A traditional server with all jar files in the distribution - A server that uses maven coordinates in module.xml files, with all artifacts downloaded as part of the provisioning process - As above, but with artifacts being lazily loaded as needed (not recommended for production, but I think this may be useful from a developer point of view) The provisioning tool will work from an XML descriptor that describes the server that is to be built. In general this information will include: - GAV of the feature packs to use - Filtering information if not all features from a pack are required (e.g. just give me JAX-RS from the EE pack. In this case the only modules/subsystems installed from the pack will be modules and subystem that JAX-RS requires). - Version overrides (e.g. give me Reaseasy 3.0.10 instead of 3.0.8), which will allow community users to easily upgrade individual components. - Configuration changes that are required (e.g. some way to add a datasource to the assembled server). The actual form this will take still needs to be decided. Note that this need to work on both a user level (a user adding a datasource) and a feature pack level (e.g. the JON feature packing adding a required data source). - GAV of deployments to install in the server. This should allow a server complete with deployments and the necessary config to be assembled and be immediately ready to be put into service. Note that if you just want a full WF install you should be able to provision it with a single line in the provisioning file, by specifying the dist feature pack. We will still provide our traditional download, which will be build by the provisioning tool as part of our build process. The provisioning tool will also be able to upgrade servers, which basically consists of provisioning a new modules directory. Rollback is provided by provisioning from an earlier version of provisioning file. When a server is provisioned the tool will make a backup copy of the file used, so it should always be possible to examine the provisioning file that was used to build the current server config. Note that when an update is performed on an existing server config will not be updated, unless the update adds an additional config file, in which case the new config file will be generated (however existing config will not be touched). Note that as a result of this split we will need to do much more frequent releases of the individual feature packs, to allow the most recent code to be integrated into dist. Implementation Plan The above changes are obviously a big job, and will not happen overnight. They are also highly likely to conflict with other changes, so maintaining a long running branch that gets rebased is not a practical option. Instead the plan it to perform the split in incremental changes. The basic steps are listed below, some of which can be performed in parallel. 1) Using the initial implementation of my build plugin (in my wildfly-build-plugin branch) we split up the server along the lines above. The code will all stay in the same repo, however the plugin will be used to build all the individual pieces, which are then assembled as part of the final build process. Note that the plugin in its current form does both the build and provision step, and the pack format is produces is far from the final pack format that we will want to use. 2) Split up the test suite into modules based on the features that they test. This will result in several smaller modules in place of a single large one, which should also be a usability improvement as individual tests will be be faster to run, and run times for all tests in a module should be more manageable. 3) Split the core into into own module. 4) Split everything else into its own module. As part of this step we need to make sure we still have the ability to run all tests against the full server, as well as against the cut down feature pack version of the server. 5) Focus on the build an provisioning tool, to implement all the features above, and to finalize the WF pack format. I think that just about covers it. There are still lots of nitty gritty details that need to be worked out, however I think this covers all the main aspects of the design. We are planning on starting work on this basically immediately, as we want to get this implemented as early in the WF9 cycle as possible. Stuart