This design proposal covers the inter related tasks of splitting up the
build, and also creating a build/provisioning system that will make it
easy for end users to consume Wildfly. Apologies for the length, but it
is a complex topic. The first part explains what we are trying to
achieve, the second part covers how we are planning to actually
implement it.

The Wildfly code base is over a million lines of java and has a test
suite that generally takes close to two hours to run in its entirety.
This makes the project very unwieldily, and the large size and slow test
suite makes development painful.

To deal with this issue we are going to split the Wildfly code base into
smaller discrete repositories. The planned split is as follows:

- Core: just the WF core
- Arquillian: the arquillian adaptors
- Servlet: a WF distribution with just Undertow, and some basic EE
functionality such as naming
- EE: All the core EE related functionality, EJB's, messaging etc
- Clustering: The core clustering functionality
- Console: The management console
- Dist: brings all the pieces together, and allows us to run all tests
against a full server

Note that this list is in no way final, and is open to debate. We will
most likely want to split up the EE component at some point, possibly
along some kind of web profile/full profile type split.

Each of these repos will build a feature pack, which will contain the
following:

- Feature specification / description
- Core version requirements (e.g. WF10)
- Dependency info on other features (e.g. RestEASY X requires CDI 1.1)
- module.xml files for all required modules that are not provided by
other features
- References to maven GAV's for jars (possibly a level of indirection
here, module.xml may just contain the group and artifact, and the
version may be in a version.properties file to allow it to be easily
overridden)
- Default configuration snippet, subsystem snippets are packaged in the
subsystem jars, templates that combine them into config files are part
of the feature pack.
- Misc files (e.g. xsds) with indication of where on path to place them

Note that a feature pack is not a complete server, it cannot simply be
extracted and run, it first needs to be assembled into a server by the
provisioning tool. The feature packs also just contain references to the
maven GAV of required jars, they do not have the actual jars in the pack
(which should make them very lightweight).

Feature packs will be assembled by the WF build tool, which is just a
maven plugin that will replace our existing hacky collection of ant
scripts.

Actual server instances will be assembled by the provisioning tool,
which will be implemented as a library with several different front
ends, including a maven plugin and a CLI (possibly integrated into our
existing CLI). In general the provisioning tool will be able to
provision three different types of servers:

- A traditional server with all jar files in the distribution
- A server that uses maven coordinates in module.xml files, with all
artifacts downloaded as part of the provisioning process
- As above, but with artifacts being lazily loaded as needed (not
recommended for production, but I think this may be useful from a
developer point of view)

The provisioning tool will work from an XML descriptor that describes
the server that is to be built. In general this information will include:

- GAV of the feature packs to use
- Filtering information if not all features from a pack are required
(e.g. just give me JAX-RS from the EE pack. In this case the only
modules/subsystems installed from the pack will be modules and subystem
that JAX-RS requires).
- Version overrides (e.g. give me Reaseasy 3.0.10 instead of 3.0.8),
which will allow community users to easily upgrade individual components.
- Configuration changes that are required (e.g. some way to add a
datasource to the assembled server). The actual form this will take
still needs to be decided. Note that this need to work on both a user
level (a user adding a datasource) and a feature pack level (e.g. the
JON feature packing adding a required data source).
- GAV of deployments to install in the server. This should allow a
server complete with deployments and the necessary config to be
assembled and be immediately ready to be put into service.

Note that if you just want a full WF install you should be able to
provision it with a single line in the provisioning file, by specifying
the dist feature pack. We will still provide our traditional download,
which will be build by the provisioning tool as part of our build process.

The provisioning tool will also be able to upgrade servers, which
basically consists of provisioning a new modules directory. Rollback is
provided by provisioning from an earlier version of provisioning file.
When a server is provisioned the tool will make a backup copy of the
file used, so it should always be possible to examine the provisioning
file that was used to build the current server config.

Note that when an update is performed on an existing server config will
not be updated, unless the update adds an additional config file, in
which case the new config file will be generated (however existing
config will not be touched).

Note that as a result of this split we will need to do much more
frequent releases of the individual feature packs, to allow the most
recent code to be integrated into dist.

Implementation Plan

The above changes are obviously a big job, and will not happen
overnight. They are also highly likely to conflict with other changes,
so maintaining a long running branch that gets rebased is not a
practical option. Instead the plan it to perform the split in
incremental changes. The basic steps are listed below, some of which can
be performed in parallel.

1) Using the initial implementation of my build plugin (in my
wildfly-build-plugin branch) we split up the server along the lines
above. The code will all stay in the same repo, however the plugin will
be used to build all the individual pieces, which are then assembled as
part of the final build process. Note that the plugin in its current
form does both the build and provision step, and the pack format is
produces is far from the final pack format that we will want to use.

2) Split up the test suite into modules based on the features that they
test. This will result in several smaller modules in place of a single
large one, which should also be a usability improvement as individual
tests will be be faster to run, and run times for all tests in a module
should be more manageable.

3) Split the core into into own module.

4) Split everything else into its own module. As part of this step we
need to make sure we still have the ability to run all tests against the
full server, as well as against the cut down feature pack version of the
server.

5) Focus on the build an provisioning tool, to implement all the
features above, and to finalize the WF pack format.

I think that just about covers it. There are still lots of nitty gritty
details that need to be worked out, however I think this covers all the
main aspects of the design. We are planning on starting work on this
basically immediately, as we want to get this implemented as early in
the WF9 cycle as possible.

Stuart









_______________________________________________
wildfly-dev mailing list
wildfly-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/wildfly-dev