[wildfly-dev] Design Proposal: Build split and provisioning

Wed Jun 11 10:59:10 EDT 2014

Peter Cai wrote:
> Hi Stuart,
> Many good points.
>
> I have some questions in this regard.
> 1, Could you please clarify the difference between feature packs,
> module, and subsystems?
> If I understand correctly, that subsystems, and module to feature packs
> is like what ingredients to recipes.

A feature pack contains subsystems and other modules.

>
> 2, To my gut feel, the feature packs you described shares the same
> concept of feature of Karaf. For more info, please refer to
> http://karaf.apache.org/manual/latest/users-guide/provisioning.html. Is
> it possible for Wildfly to be built on top of OSGi container in the future?
>

It is kinda like Karaf, but we are not based on OSGi, and have no plans 
to move.

Stuart

> Regards,
> Peter C
>
>
> On Wed, Jun 11, 2014 at 1:13 AM, Stuart Douglas <sdouglas at redhat.com
> <mailto:sdouglas at redhat.com>> wrote:
>
>     This design proposal covers the inter related tasks of splitting up the
>     build, and also creating a build/provisioning system that will make it
>     easy for end users to consume Wildfly. Apologies for the length, but it
>     is a complex topic. The first part explains what we are trying to
>     achieve, the second part covers how we are planning to actually
>     implement it.
>
>     The Wildfly code base is over a million lines of java and has a test
>     suite that generally takes close to two hours to run in its entirety.
>     This makes the project very unwieldily, and the large size and slow test
>     suite makes development painful.
>
>     To deal with this issue we are going to split the Wildfly code base into
>     smaller discrete repositories. The planned split is as follows:
>
>     - Core: just the WF core
>     - Arquillian: the arquillian adaptors
>     - Servlet: a WF distribution with just Undertow, and some basic EE
>     functionality such as naming
>     - EE: All the core EE related functionality, EJB's, messaging etc
>     - Clustering: The core clustering functionality
>     - Console: The management console
>     - Dist: brings all the pieces together, and allows us to run all tests
>     against a full server
>
>     Note that this list is in no way final, and is open to debate. We will
>     most likely want to split up the EE component at some point, possibly
>     along some kind of web profile/full profile type split.
>
>     Each of these repos will build a feature pack, which will contain the
>     following:
>
>     - Feature specification / description
>     - Core version requirements (e.g. WF10)
>     - Dependency info on other features (e.g. RestEASY X requires CDI 1.1)
>     - module.xml files for all required modules that are not provided by
>     other features
>     - References to maven GAV's for jars (possibly a level of indirection
>     here, module.xml may just contain the group and artifact, and the
>     version may be in a version.properties file to allow it to be easily
>     overridden)
>     - Default configuration snippet, subsystem snippets are packaged in the
>     subsystem jars, templates that combine them into config files are part
>     of the feature pack.
>     - Misc files (e.g. xsds) with indication of where on path to place them
>
>     Note that a feature pack is not a complete server, it cannot simply be
>     extracted and run, it first needs to be assembled into a server by the
>     provisioning tool. The feature packs also just contain references to the
>     maven GAV of required jars, they do not have the actual jars in the pack
>     (which should make them very lightweight).
>
>     Feature packs will be assembled by the WF build tool, which is just a
>     maven plugin that will replace our existing hacky collection of ant
>     scripts.
>
>     Actual server instances will be assembled by the provisioning tool,
>     which will be implemented as a library with several different front
>     ends, including a maven plugin and a CLI (possibly integrated into our
>     existing CLI). In general the provisioning tool will be able to
>     provision three different types of servers:
>
>     - A traditional server with all jar files in the distribution
>     - A server that uses maven coordinates in module.xml files, with all
>     artifacts downloaded as part of the provisioning process
>     - As above, but with artifacts being lazily loaded as needed (not
>     recommended for production, but I think this may be useful from a
>     developer point of view)
>
>     The provisioning tool will work from an XML descriptor that describes
>     the server that is to be built. In general this information will
>     include:
>
>     - GAV of the feature packs to use
>     - Filtering information if not all features from a pack are required
>     (e.g. just give me JAX-RS from the EE pack. In this case the only
>     modules/subsystems installed from the pack will be modules and subystem
>     that JAX-RS requires).
>     - Version overrides (e.g. give me Reaseasy 3.0.10 instead of 3.0.8),
>     which will allow community users to easily upgrade individual
>     components.
>     - Configuration changes that are required (e.g. some way to add a
>     datasource to the assembled server). The actual form this will take
>     still needs to be decided. Note that this need to work on both a user
>     level (a user adding a datasource) and a feature pack level (e.g. the
>     JON feature packing adding a required data source).
>     - GAV of deployments to install in the server. This should allow a
>     server complete with deployments and the necessary config to be
>     assembled and be immediately ready to be put into service.
>
>     Note that if you just want a full WF install you should be able to
>     provision it with a single line in the provisioning file, by specifying
>     the dist feature pack. We will still provide our traditional download,
>     which will be build by the provisioning tool as part of our build
>     process.
>
>     The provisioning tool will also be able to upgrade servers, which
>     basically consists of provisioning a new modules directory. Rollback is
>     provided by provisioning from an earlier version of provisioning file.
>     When a server is provisioned the tool will make a backup copy of the
>     file used, so it should always be possible to examine the provisioning
>     file that was used to build the current server config.
>
>     Note that when an update is performed on an existing server config will
>     not be updated, unless the update adds an additional config file, in
>     which case the new config file will be generated (however existing
>     config will not be touched).
>
>     Note that as a result of this split we will need to do much more
>     frequent releases of the individual feature packs, to allow the most
>     recent code to be integrated into dist.
>
>     Implementation Plan
>
>     The above changes are obviously a big job, and will not happen
>     overnight. They are also highly likely to conflict with other changes,
>     so maintaining a long running branch that gets rebased is not a
>     practical option. Instead the plan it to perform the split in
>     incremental changes. The basic steps are listed below, some of which can
>     be performed in parallel.
>
>     1) Using the initial implementation of my build plugin (in my
>     wildfly-build-plugin branch) we split up the server along the lines
>     above. The code will all stay in the same repo, however the plugin will
>     be used to build all the individual pieces, which are then assembled as
>     part of the final build process. Note that the plugin in its current
>     form does both the build and provision step, and the pack format is
>     produces is far from the final pack format that we will want to use.
>
>     2) Split up the test suite into modules based on the features that they
>     test. This will result in several smaller modules in place of a single
>     large one, which should also be a usability improvement as individual
>     tests will be be faster to run, and run times for all tests in a module
>     should be more manageable.
>
>     3) Split the core into into own module.
>
>     4) Split everything else into its own module. As part of this step we
>     need to make sure we still have the ability to run all tests against the
>     full server, as well as against the cut down feature pack version of the
>     server.
>
>     5) Focus on the build an provisioning tool, to implement all the
>     features above, and to finalize the WF pack format.
>
>     I think that just about covers it. There are still lots of nitty gritty
>     details that need to be worked out, however I think this covers all the
>     main aspects of the design. We are planning on starting work on this
>     basically immediately, as we want to get this implemented as early in
>     the WF9 cycle as possible.
>
>     Stuart
>
>
>
>
>
>
>
>
>
>     _______________________________________________
>     wildfly-dev mailing list
>     wildfly-dev at lists.jboss.org <mailto:wildfly-dev at lists.jboss.org>
>     https://lists.jboss.org/mailman/listinfo/wildfly-dev
>
>
> _______________________________________________
> wildfly-dev mailing list
> wildfly-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/wildfly-dev