[wildfly-dev] Design Proposal: Build split and provisioning

[Stuart Douglas]

Just double checked this and it works fine. Basically I just created a 
core distro using the script in my wildfly-build-plugin branch, and then 
initialized a new git repo with the result.

I then added a new commit, and tested cherry-picking it onto an older 
code base, and it worked as expected.

Stuart

Darran Lofthouse wrote:
> On 12/06/14 15:48, Stuart Douglas wrote:
>> I am not sure if either of these choices will make much difference? In
>> general our directory layout will be the same, so it should just be
>> possibly to cherry pick patches from WF.
>
> That is probably true but I think this is an important enough point that
> before changes are made we know that it will be possible rather than
> than just should be possible.
>
>> Stuart
>>
>> Darran Lofthouse wrote:
>>> Please also consider the requirements we need to follow when maintaining
>>> the products based on previous AS7/WildFly releases.
>>>
>>> Regards,
>>> Darran Lofthouse.
>>>
>>>
>>> On 11/06/14 16:57, Stuart Douglas wrote:
>>>> Something that I did not cover was how to actually do the split it
>>>> terms
>>>> of preserving history. We have a few options:
>>>>
>>>> 1) Just copy the files into a clean repo. There is no history in the
>>>> repo, but you could always check the existing wildfly repo if you
>>>> really
>>>> need it.
>>>>
>>>> 2) Copy the complete WF repo and then delete the parts that are not
>>>> going to be part of the new repo. This leaves complete history, but
>>>> means that the check outs will be larger than they should be.
>>>>
>>>> 3) Use git-filter-branch to create a new repo with just the history of
>>>> the relevant files. We still have a small checkout size, but the
>>>> history
>>>> is still in the repo.
>>>>
>>>> I think we should go with option 3.
>>>>
>>>> Stuart
>>>>
>>>> Stuart Douglas wrote:
>>>>> This design proposal covers the inter related tasks of splitting up
>>>>> the
>>>>> build, and also creating a build/provisioning system that will make it
>>>>> easy for end users to consume Wildfly. Apologies for the length,
>>>>> but it
>>>>> is a complex topic. The first part explains what we are trying to
>>>>> achieve, the second part covers how we are planning to actually
>>>>> implement it.
>>>>>
>>>>> The Wildfly code base is over a million lines of java and has a test
>>>>> suite that generally takes close to two hours to run in its entirety.
>>>>> This makes the project very unwieldily, and the large size and slow
>>>>> test
>>>>> suite makes development painful.
>>>>>
>>>>> To deal with this issue we are going to split the Wildfly code base
>>>>> into
>>>>> smaller discrete repositories. The planned split is as follows:
>>>>>
>>>>> - Core: just the WF core
>>>>> - Arquillian: the arquillian adaptors
>>>>> - Servlet: a WF distribution with just Undertow, and some basic EE
>>>>> functionality such as naming
>>>>> - EE: All the core EE related functionality, EJB's, messaging etc
>>>>> - Clustering: The core clustering functionality
>>>>> - Console: The management console
>>>>> - Dist: brings all the pieces together, and allows us to run all tests
>>>>> against a full server
>>>>>
>>>>> Note that this list is in no way final, and is open to debate. We will
>>>>> most likely want to split up the EE component at some point, possibly
>>>>> along some kind of web profile/full profile type split.
>>>>>
>>>>> Each of these repos will build a feature pack, which will contain the
>>>>> following:
>>>>>
>>>>> - Feature specification / description
>>>>> - Core version requirements (e.g. WF10)
>>>>> - Dependency info on other features (e.g. RestEASY X requires CDI 1.1)
>>>>> - module.xml files for all required modules that are not provided by
>>>>> other features
>>>>> - References to maven GAV's for jars (possibly a level of indirection
>>>>> here, module.xml may just contain the group and artifact, and the
>>>>> version may be in a version.properties file to allow it to be easily
>>>>> overridden)
>>>>> - Default configuration snippet, subsystem snippets are packaged in
>>>>> the
>>>>> subsystem jars, templates that combine them into config files are part
>>>>> of the feature pack.
>>>>> - Misc files (e.g. xsds) with indication of where on path to place
>>>>> them
>>>>>
>>>>> Note that a feature pack is not a complete server, it cannot simply be
>>>>> extracted and run, it first needs to be assembled into a server by the
>>>>> provisioning tool. The feature packs also just contain references to
>>>>> the
>>>>> maven GAV of required jars, they do not have the actual jars in the
>>>>> pack
>>>>> (which should make them very lightweight).
>>>>>
>>>>> Feature packs will be assembled by the WF build tool, which is just a
>>>>> maven plugin that will replace our existing hacky collection of ant
>>>>> scripts.
>>>>>
>>>>> Actual server instances will be assembled by the provisioning tool,
>>>>> which will be implemented as a library with several different front
>>>>> ends, including a maven plugin and a CLI (possibly integrated into our
>>>>> existing CLI). In general the provisioning tool will be able to
>>>>> provision three different types of servers:
>>>>>
>>>>> - A traditional server with all jar files in the distribution
>>>>> - A server that uses maven coordinates in module.xml files, with all
>>>>> artifacts downloaded as part of the provisioning process
>>>>> - As above, but with artifacts being lazily loaded as needed (not
>>>>> recommended for production, but I think this may be useful from a
>>>>> developer point of view)
>>>>>
>>>>> The provisioning tool will work from an XML descriptor that describes
>>>>> the server that is to be built. In general this information will
>>>>> include:
>>>>>
>>>>> - GAV of the feature packs to use
>>>>> - Filtering information if not all features from a pack are required
>>>>> (e.g. just give me JAX-RS from the EE pack. In this case the only
>>>>> modules/subsystems installed from the pack will be modules and
>>>>> subystem
>>>>> that JAX-RS requires).
>>>>> - Version overrides (e.g. give me Reaseasy 3.0.10 instead of 3.0.8),
>>>>> which will allow community users to easily upgrade individual
>>>>> components.
>>>>> - Configuration changes that are required (e.g. some way to add a
>>>>> datasource to the assembled server). The actual form this will take
>>>>> still needs to be decided. Note that this need to work on both a user
>>>>> level (a user adding a datasource) and a feature pack level (e.g. the
>>>>> JON feature packing adding a required data source).
>>>>> - GAV of deployments to install in the server. This should allow a
>>>>> server complete with deployments and the necessary config to be
>>>>> assembled and be immediately ready to be put into service.
>>>>>
>>>>> Note that if you just want a full WF install you should be able to
>>>>> provision it with a single line in the provisioning file, by
>>>>> specifying
>>>>> the dist feature pack. We will still provide our traditional download,
>>>>> which will be build by the provisioning tool as part of our build
>>>>> process.
>>>>>
>>>>> The provisioning tool will also be able to upgrade servers, which
>>>>> basically consists of provisioning a new modules directory.
>>>>> Rollback is
>>>>> provided by provisioning from an earlier version of provisioning file.
>>>>> When a server is provisioned the tool will make a backup copy of the
>>>>> file used, so it should always be possible to examine the provisioning
>>>>> file that was used to build the current server config.
>>>>>
>>>>> Note that when an update is performed on an existing server config
>>>>> will
>>>>> not be updated, unless the update adds an additional config file, in
>>>>> which case the new config file will be generated (however existing
>>>>> config will not be touched).
>>>>>
>>>>> Note that as a result of this split we will need to do much more
>>>>> frequent releases of the individual feature packs, to allow the most
>>>>> recent code to be integrated into dist.
>>>>>
>>>>> Implementation Plan
>>>>>
>>>>> The above changes are obviously a big job, and will not happen
>>>>> overnight. They are also highly likely to conflict with other changes,
>>>>> so maintaining a long running branch that gets rebased is not a
>>>>> practical option. Instead the plan it to perform the split in
>>>>> incremental changes. The basic steps are listed below, some of which
>>>>> can
>>>>> be performed in parallel.
>>>>>
>>>>> 1) Using the initial implementation of my build plugin (in my
>>>>> wildfly-build-plugin branch) we split up the server along the lines
>>>>> above. The code will all stay in the same repo, however the plugin
>>>>> will
>>>>> be used to build all the individual pieces, which are then
>>>>> assembled as
>>>>> part of the final build process. Note that the plugin in its current
>>>>> form does both the build and provision step, and the pack format is
>>>>> produces is far from the final pack format that we will want to use.
>>>>>
>>>>> 2) Split up the test suite into modules based on the features that
>>>>> they
>>>>> test. This will result in several smaller modules in place of a single
>>>>> large one, which should also be a usability improvement as individual
>>>>> tests will be be faster to run, and run times for all tests in a
>>>>> module
>>>>> should be more manageable.
>>>>>
>>>>> 3) Split the core into into own module.
>>>>>
>>>>> 4) Split everything else into its own module. As part of this step we
>>>>> need to make sure we still have the ability to run all tests against
>>>>> the
>>>>> full server, as well as against the cut down feature pack version of
>>>>> the
>>>>> server.
>>>>>
>>>>> 5) Focus on the build an provisioning tool, to implement all the
>>>>> features above, and to finalize the WF pack format.
>>>>>
>>>>> I think that just about covers it. There are still lots of nitty
>>>>> gritty
>>>>> details that need to be worked out, however I think this covers all
>>>>> the
>>>>> main aspects of the design. We are planning on starting work on this
>>>>> basically immediately, as we want to get this implemented as early in
>>>>> the WF9 cycle as possible.
>>>>>
>>>>> Stuart
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> wildfly-dev mailing list
>>>>> wildfly-dev at lists.jboss.org
>>>>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>>>> _______________________________________________
>>>> wildfly-dev mailing list
>>>> wildfly-dev at lists.jboss.org
>>>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>>>>
>>> _______________________________________________
>>> wildfly-dev mailing list
>>> wildfly-dev at lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/wildfly-dev