[wildfly-dev] Design Proposal: Build split and provisioning

[Darran Lofthouse]

On 12/06/14 15:48, Stuart Douglas wrote:
> I am not sure if either of these choices will make much difference? In
> general our directory layout will be the same, so it should just be
> possibly to cherry pick patches from WF.

That is probably true but I think this is an important enough point that 
before changes are made we know that it will be possible rather than 
than just should be possible.

> Stuart
>
> Darran Lofthouse wrote:
>> Please also consider the requirements we need to follow when maintaining
>> the products based on previous AS7/WildFly releases.
>>
>> Regards,
>> Darran Lofthouse.
>>
>>
>> On 11/06/14 16:57, Stuart Douglas wrote:
>>> Something that I did not cover was how to actually do the split it terms
>>> of preserving history. We have a few options:
>>>
>>> 1) Just copy the files into a clean repo. There is no history in the
>>> repo, but you could always check the existing wildfly repo if you really
>>> need it.
>>>
>>> 2) Copy the complete WF repo and then delete the parts that are not
>>> going to be part of the new repo. This leaves complete history, but
>>> means that the check outs will be larger than they should be.
>>>
>>> 3) Use git-filter-branch to create a new repo with just the history of
>>> the relevant files. We still have a small checkout size, but the history
>>> is still in the repo.
>>>
>>> I think we should go with option 3.
>>>
>>> Stuart
>>>
>>> Stuart Douglas wrote:
>>>> This design proposal covers the inter related tasks of splitting up the
>>>> build, and also creating a build/provisioning system that will make it
>>>> easy for end users to consume Wildfly. Apologies for the length, but it
>>>> is a complex topic. The first part explains what we are trying to
>>>> achieve, the second part covers how we are planning to actually
>>>> implement it.
>>>>
>>>> The Wildfly code base is over a million lines of java and has a test
>>>> suite that generally takes close to two hours to run in its entirety.
>>>> This makes the project very unwieldily, and the large size and slow
>>>> test
>>>> suite makes development painful.
>>>>
>>>> To deal with this issue we are going to split the Wildfly code base
>>>> into
>>>> smaller discrete repositories. The planned split is as follows:
>>>>
>>>> - Core: just the WF core
>>>> - Arquillian: the arquillian adaptors
>>>> - Servlet: a WF distribution with just Undertow, and some basic EE
>>>> functionality such as naming
>>>> - EE: All the core EE related functionality, EJB's, messaging etc
>>>> - Clustering: The core clustering functionality
>>>> - Console: The management console
>>>> - Dist: brings all the pieces together, and allows us to run all tests
>>>> against a full server
>>>>
>>>> Note that this list is in no way final, and is open to debate. We will
>>>> most likely want to split up the EE component at some point, possibly
>>>> along some kind of web profile/full profile type split.
>>>>
>>>> Each of these repos will build a feature pack, which will contain the
>>>> following:
>>>>
>>>> - Feature specification / description
>>>> - Core version requirements (e.g. WF10)
>>>> - Dependency info on other features (e.g. RestEASY X requires CDI 1.1)
>>>> - module.xml files for all required modules that are not provided by
>>>> other features
>>>> - References to maven GAV's for jars (possibly a level of indirection
>>>> here, module.xml may just contain the group and artifact, and the
>>>> version may be in a version.properties file to allow it to be easily
>>>> overridden)
>>>> - Default configuration snippet, subsystem snippets are packaged in the
>>>> subsystem jars, templates that combine them into config files are part
>>>> of the feature pack.
>>>> - Misc files (e.g. xsds) with indication of where on path to place them
>>>>
>>>> Note that a feature pack is not a complete server, it cannot simply be
>>>> extracted and run, it first needs to be assembled into a server by the
>>>> provisioning tool. The feature packs also just contain references to
>>>> the
>>>> maven GAV of required jars, they do not have the actual jars in the
>>>> pack
>>>> (which should make them very lightweight).
>>>>
>>>> Feature packs will be assembled by the WF build tool, which is just a
>>>> maven plugin that will replace our existing hacky collection of ant
>>>> scripts.
>>>>
>>>> Actual server instances will be assembled by the provisioning tool,
>>>> which will be implemented as a library with several different front
>>>> ends, including a maven plugin and a CLI (possibly integrated into our
>>>> existing CLI). In general the provisioning tool will be able to
>>>> provision three different types of servers:
>>>>
>>>> - A traditional server with all jar files in the distribution
>>>> - A server that uses maven coordinates in module.xml files, with all
>>>> artifacts downloaded as part of the provisioning process
>>>> - As above, but with artifacts being lazily loaded as needed (not
>>>> recommended for production, but I think this may be useful from a
>>>> developer point of view)
>>>>
>>>> The provisioning tool will work from an XML descriptor that describes
>>>> the server that is to be built. In general this information will
>>>> include:
>>>>
>>>> - GAV of the feature packs to use
>>>> - Filtering information if not all features from a pack are required
>>>> (e.g. just give me JAX-RS from the EE pack. In this case the only
>>>> modules/subsystems installed from the pack will be modules and subystem
>>>> that JAX-RS requires).
>>>> - Version overrides (e.g. give me Reaseasy 3.0.10 instead of 3.0.8),
>>>> which will allow community users to easily upgrade individual
>>>> components.
>>>> - Configuration changes that are required (e.g. some way to add a
>>>> datasource to the assembled server). The actual form this will take
>>>> still needs to be decided. Note that this need to work on both a user
>>>> level (a user adding a datasource) and a feature pack level (e.g. the
>>>> JON feature packing adding a required data source).
>>>> - GAV of deployments to install in the server. This should allow a
>>>> server complete with deployments and the necessary config to be
>>>> assembled and be immediately ready to be put into service.
>>>>
>>>> Note that if you just want a full WF install you should be able to
>>>> provision it with a single line in the provisioning file, by specifying
>>>> the dist feature pack. We will still provide our traditional download,
>>>> which will be build by the provisioning tool as part of our build
>>>> process.
>>>>
>>>> The provisioning tool will also be able to upgrade servers, which
>>>> basically consists of provisioning a new modules directory. Rollback is
>>>> provided by provisioning from an earlier version of provisioning file.
>>>> When a server is provisioned the tool will make a backup copy of the
>>>> file used, so it should always be possible to examine the provisioning
>>>> file that was used to build the current server config.
>>>>
>>>> Note that when an update is performed on an existing server config will
>>>> not be updated, unless the update adds an additional config file, in
>>>> which case the new config file will be generated (however existing
>>>> config will not be touched).
>>>>
>>>> Note that as a result of this split we will need to do much more
>>>> frequent releases of the individual feature packs, to allow the most
>>>> recent code to be integrated into dist.
>>>>
>>>> Implementation Plan
>>>>
>>>> The above changes are obviously a big job, and will not happen
>>>> overnight. They are also highly likely to conflict with other changes,
>>>> so maintaining a long running branch that gets rebased is not a
>>>> practical option. Instead the plan it to perform the split in
>>>> incremental changes. The basic steps are listed below, some of which
>>>> can
>>>> be performed in parallel.
>>>>
>>>> 1) Using the initial implementation of my build plugin (in my
>>>> wildfly-build-plugin branch) we split up the server along the lines
>>>> above. The code will all stay in the same repo, however the plugin will
>>>> be used to build all the individual pieces, which are then assembled as
>>>> part of the final build process. Note that the plugin in its current
>>>> form does both the build and provision step, and the pack format is
>>>> produces is far from the final pack format that we will want to use.
>>>>
>>>> 2) Split up the test suite into modules based on the features that they
>>>> test. This will result in several smaller modules in place of a single
>>>> large one, which should also be a usability improvement as individual
>>>> tests will be be faster to run, and run times for all tests in a module
>>>> should be more manageable.
>>>>
>>>> 3) Split the core into into own module.
>>>>
>>>> 4) Split everything else into its own module. As part of this step we
>>>> need to make sure we still have the ability to run all tests against
>>>> the
>>>> full server, as well as against the cut down feature pack version of
>>>> the
>>>> server.
>>>>
>>>> 5) Focus on the build an provisioning tool, to implement all the
>>>> features above, and to finalize the WF pack format.
>>>>
>>>> I think that just about covers it. There are still lots of nitty gritty
>>>> details that need to be worked out, however I think this covers all the
>>>> main aspects of the design. We are planning on starting work on this
>>>> basically immediately, as we want to get this implemented as early in
>>>> the WF9 cycle as possible.
>>>>
>>>> Stuart
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> wildfly-dev mailing list
>>>> wildfly-dev at lists.jboss.org
>>>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>>> _______________________________________________
>>> wildfly-dev mailing list
>>> wildfly-dev at lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>>>
>> _______________________________________________
>> wildfly-dev mailing list
>> wildfly-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/wildfly-dev