[wildfly-dev] Design Proposal: Build split and provisioning

Thu Jun 12 06:33:26 EDT 2014

Hi,
Concerning tests and the integration tests in the test suite, will the splitting requires them to be run twice or more ?
Let's say we test CDI integration with JAX-RS, will it be tested in the feature pack JAR-RS, or in JAX-RS and JavaEE and complete dist ?
Cheers,
Emmanuel

Le 11/06/2014 17:57, Stuart Douglas a écrit :
> Something that I did not cover was how to actually do the split it terms 
> of preserving history. We have a few options:
> 
> 1) Just copy the files into a clean repo. There is no history in the 
> repo, but you could always check the existing wildfly repo if you really 
> need it.
> 
> 2) Copy the complete WF repo and then delete the parts that are not 
> going to be part of the new repo. This leaves complete history, but 
> means that the check outs will be larger than they should be.
> 
> 3) Use git-filter-branch to create a new repo with just the history of 
> the relevant files. We still have a small checkout size, but the history 
> is still in the repo.
> 
> I think we should go with option 3.
> 
> Stuart
> 
> Stuart Douglas wrote:
>> This design proposal covers the inter related tasks of splitting up the
>> build, and also creating a build/provisioning system that will make it
>> easy for end users to consume Wildfly. Apologies for the length, but it
>> is a complex topic. The first part explains what we are trying to
>> achieve, the second part covers how we are planning to actually
>> implement it.
>>
>> The Wildfly code base is over a million lines of java and has a test
>> suite that generally takes close to two hours to run in its entirety.
>> This makes the project very unwieldily, and the large size and slow test
>> suite makes development painful.
>>
>> To deal with this issue we are going to split the Wildfly code base into
>> smaller discrete repositories. The planned split is as follows:
>>
>> - Core: just the WF core
>> - Arquillian: the arquillian adaptors
>> - Servlet: a WF distribution with just Undertow, and some basic EE
>> functionality such as naming
>> - EE: All the core EE related functionality, EJB's, messaging etc
>> - Clustering: The core clustering functionality
>> - Console: The management console
>> - Dist: brings all the pieces together, and allows us to run all tests
>> against a full server
>>
>> Note that this list is in no way final, and is open to debate. We will
>> most likely want to split up the EE component at some point, possibly
>> along some kind of web profile/full profile type split.
>>
>> Each of these repos will build a feature pack, which will contain the
>> following:
>>
>> - Feature specification / description
>> - Core version requirements (e.g. WF10)
>> - Dependency info on other features (e.g. RestEASY X requires CDI 1.1)
>> - module.xml files for all required modules that are not provided by
>> other features
>> - References to maven GAV's for jars (possibly a level of indirection
>> here, module.xml may just contain the group and artifact, and the
>> version may be in a version.properties file to allow it to be easily
>> overridden)
>> - Default configuration snippet, subsystem snippets are packaged in the
>> subsystem jars, templates that combine them into config files are part
>> of the feature pack.
>> - Misc files (e.g. xsds) with indication of where on path to place them
>>
>> Note that a feature pack is not a complete server, it cannot simply be
>> extracted and run, it first needs to be assembled into a server by the
>> provisioning tool. The feature packs also just contain references to the
>> maven GAV of required jars, they do not have the actual jars in the pack
>> (which should make them very lightweight).
>>
>> Feature packs will be assembled by the WF build tool, which is just a
>> maven plugin that will replace our existing hacky collection of ant
>> scripts.
>>
>> Actual server instances will be assembled by the provisioning tool,
>> which will be implemented as a library with several different front
>> ends, including a maven plugin and a CLI (possibly integrated into our
>> existing CLI). In general the provisioning tool will be able to
>> provision three different types of servers:
>>
>> - A traditional server with all jar files in the distribution
>> - A server that uses maven coordinates in module.xml files, with all
>> artifacts downloaded as part of the provisioning process
>> - As above, but with artifacts being lazily loaded as needed (not
>> recommended for production, but I think this may be useful from a
>> developer point of view)
>>
>> The provisioning tool will work from an XML descriptor that describes
>> the server that is to be built. In general this information will include:
>>
>> - GAV of the feature packs to use
>> - Filtering information if not all features from a pack are required
>> (e.g. just give me JAX-RS from the EE pack. In this case the only
>> modules/subsystems installed from the pack will be modules and subystem
>> that JAX-RS requires).
>> - Version overrides (e.g. give me Reaseasy 3.0.10 instead of 3.0.8),
>> which will allow community users to easily upgrade individual components.
>> - Configuration changes that are required (e.g. some way to add a
>> datasource to the assembled server). The actual form this will take
>> still needs to be decided. Note that this need to work on both a user
>> level (a user adding a datasource) and a feature pack level (e.g. the
>> JON feature packing adding a required data source).
>> - GAV of deployments to install in the server. This should allow a
>> server complete with deployments and the necessary config to be
>> assembled and be immediately ready to be put into service.
>>
>> Note that if you just want a full WF install you should be able to
>> provision it with a single line in the provisioning file, by specifying
>> the dist feature pack. We will still provide our traditional download,
>> which will be build by the provisioning tool as part of our build process.
>>
>> The provisioning tool will also be able to upgrade servers, which
>> basically consists of provisioning a new modules directory. Rollback is
>> provided by provisioning from an earlier version of provisioning file.
>> When a server is provisioned the tool will make a backup copy of the
>> file used, so it should always be possible to examine the provisioning
>> file that was used to build the current server config.
>>
>> Note that when an update is performed on an existing server config will
>> not be updated, unless the update adds an additional config file, in
>> which case the new config file will be generated (however existing
>> config will not be touched).
>>
>> Note that as a result of this split we will need to do much more
>> frequent releases of the individual feature packs, to allow the most
>> recent code to be integrated into dist.
>>
>> Implementation Plan
>>
>> The above changes are obviously a big job, and will not happen
>> overnight. They are also highly likely to conflict with other changes,
>> so maintaining a long running branch that gets rebased is not a
>> practical option. Instead the plan it to perform the split in
>> incremental changes. The basic steps are listed below, some of which can
>> be performed in parallel.
>>
>> 1) Using the initial implementation of my build plugin (in my
>> wildfly-build-plugin branch) we split up the server along the lines
>> above. The code will all stay in the same repo, however the plugin will
>> be used to build all the individual pieces, which are then assembled as
>> part of the final build process. Note that the plugin in its current
>> form does both the build and provision step, and the pack format is
>> produces is far from the final pack format that we will want to use.
>>
>> 2) Split up the test suite into modules based on the features that they
>> test. This will result in several smaller modules in place of a single
>> large one, which should also be a usability improvement as individual
>> tests will be be faster to run, and run times for all tests in a module
>> should be more manageable.
>>
>> 3) Split the core into into own module.
>>
>> 4) Split everything else into its own module. As part of this step we
>> need to make sure we still have the ability to run all tests against the
>> full server, as well as against the cut down feature pack version of the
>> server.
>>
>> 5) Focus on the build an provisioning tool, to implement all the
>> features above, and to finalize the WF pack format.
>>
>> I think that just about covers it. There are still lots of nitty gritty
>> details that need to be worked out, however I think this covers all the
>> main aspects of the design. We are planning on starting work on this
>> basically immediately, as we want to get this implemented as early in
>> the WF9 cycle as possible.
>>
>> Stuart
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> _______________________________________________
>> wildfly-dev mailing list
>> wildfly-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
> _______________________________________________
> wildfly-dev mailing list
> wildfly-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/wildfly-dev
> 

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 538 bytes
Desc: OpenPGP digital signature
Url : http://lists.jboss.org/pipermail/wildfly-dev/attachments/20140612/dcf87b98/attachment.bin 