Hey guys, TL;DR I've been experimenting with Alexey to update a customized provisioned server using the provisioning tool [1]. I'm using the syncing operations [2] that I created a while back by porting the domain synchronization operations to standalone (to synchronize standalone instances in a cloud environment). I'm looking for some feedback on this approach. Cheers Emmanuel [1]: https://github.com/ehsavoie/pm/tree/upgrade-feature-pack [2]: https://github.com/ehsavoie/wildfly-core/tree/model_diff ---- full version: Updating WildFly <https://github.com/ehsavoie/pm/blob/upgrade-feature-pack/ docs/guide/wildfly/upgrade.adoc#terminology>Terminology Updating is the process of applying a fix pack that will increment the micro version. There should be no compatiblity issue. Upgrading is the transition of a minor version, compatiblity should be available but there are a lot more changes. While the mecanisms discussed here are general, they might need more refinement for an upgrade. <https://github.com/ehsavoie/pm/blob/upgrade-feature-pack/ docs/guide/wildfly/upgrade.adoc#use-case>Use case The use case is quite simple: *I have version 1.0.0 installed and i want to update to 1.0.1 but I have locally customized my server and i’d like to keep those changes*. We have several local elements to take into account: * filesystem changes (files added, removed or deleted). * configuration changes. The basic idea is to diff the existing instance with a pure new installation of the same version then apply those changes to a new provisioned version instance for staging. We can keep it at the basic filesystem approach with some simple merge strategy (theirs, ours). We can use the plugin to go into more details. For exemple using the model diff between standalone Wildfly instances we can create a cli script to reconfigure the upgraded instance in a post-installation step. <https://github.com/ehsavoie/pm/blob/upgrade-feature-pack/ docs/guide/wildfly/upgrade.adoc#diffing-the-filesystem>Diffing the filesystem The idea is to compare the instance to be upgraded with one instance provisioned with the same feature packs as the one we want to upgrade. The plugin will provide a list of files or regexp to be excluded. Each file will be hashed and we compare the hash + the relative path to deduce deleted, modified or added files. For textual files we can provide a diff (and the means to apply it), but maybe that should be for a later version as some kind of interaction with the user might be required. <https://github.com/ehsavoie/pm/blob/upgrade-feature-pack/ docs/guide/wildfly/upgrade.adoc#wildfly-standalone-plugin>WildFly standalone plugin This is a specialization of the upgrading algorithm: * Filtering out some of the 'unimportant' files (tmp, logs). * Creating diff of textual files (for example the realm properties) which will be applied (merging strategy à la git). * Using an embedded standalone it creates a jboss-cli script to reconfigure the server (adding/removing extensions and reconfiguring subsystems). * Deleting files that were removed. This is done on a staging upgraded instance before being copied over the old instance. I have added a diff/sync operation in standalone that is quite similar to what happens when a slave HC connects to the DC. Thus I start the current installation, and connect to it from an embedded server using the initial configuration and diff the models. This is 'experimental' but it works nicely (I was able to 'upgrade' from the standalone.xml of wildfly-core to the standalone-full.xml of wildfly). I’m talking only about the model part, I leave the files to the filesystem 'diffing' but it wiill work with managed deployments are those are added by the filesystem part and then the deployment reference is added in the model. For a future version of the tooling/plugin we might look for a way to interract more with the user (for example for applying the textual diffs to choose what to do per file instead of globally). Also currently the filters for excluding files are defined by the plugin but we could enrich them from the tooling also. <https://github.com/ehsavoie/pm/blob/upgrade-feature-pack/ docs/guide/wildfly/upgrade.adoc#producing-an-update-feature-pack>Producing an update feature pack From the initial upgrade mecanism Alexey has seen the potential to create a feature pack instead of my initial format. Currently i’m able to create and installa a feature-pack that will supersede the initial installation with its own local modifications. Thus from my customized instance I can produce a feature pack that will allow me to reinstall the same instance. Maybe this can be also use to produce upgrade feature pack for patching. <https://github.com/ehsavoie/pm/blob/upgrade-feature-pack/ docs/guide/wildfly/upgrade.adoc#wildfly-domain-mode>WildFly domain mode Domain mode is a bit more complex, and we need to think how to manager the model changes. Those can be at the domain level or the host level and depending on the instance target we would need to get the changes from the domain.xml or/and the host.xml. I’m thinking about applying the same strategy as what was done for standalone : aka expose the sync operations to an embedded HC. _______________________________________________ wildfly-dev mailing list wildfly-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/wildfly-dev

Thanks for sharing that Emmanuel, seems to be coming along well! On Mon, Oct 2, 2017 at 9:36 AM Emmanuel Hugonnet <ehugonne(a)redhat.com&gt; wrote: > Hi, > > For those who can take 7'26 of their time to look at what we can do with > the new provisioning tool to build and share custom feature packs. > > https://www.dropbox.com/s/84133sgsjef7pqs/feature_pack.mp4?dl=0 > > > _______________________________________________ > wildfly-dev mailing list > wildfly-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/wildfly-dev _______________________________________________ wildfly-dev mailing list wildfly-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/wildfly-dev

On 25 Sep 2017, at 10:06, Emmanuel Hugonnet <ehugonne(a)redhat.com&gt; wrote: Hey guys, TL;DR I've been experimenting with Alexey to update a customized provisioned server using the provisioning tool [1]. I'm using the syncing operations [2] that I created a while back by porting the domain synchronization operations to standalone (to synchronize standalone instances in a cloud environment). I'm looking for some feedback on this approach. Cheers Emmanuel [1]: https://github.com/ehsavoie/pm/tree/upgrade-feature-pack [2]: https://github.com/ehsavoie/wildfly-core/tree/model_diff ---- full version: Updating WildFly <https://github.com/ehsavoie/pm/blob/upgrade-feature-pack/docs/guide/wildf... Updating is the process of applying a fix pack that will increment the micro version. There should be no compatiblity issue. Upgrading is the transition of a minor version, compatiblity should be available but there are a lot more changes. While the mecanisms discussed here are general, they might need more refinement for an upgrade. <https://github.com/ehsavoie/pm/blob/upgrade-feature-pack/docs/guide/wildf... case The use case is quite simple: *I have version 1.0.0 installed and i want to update to 1.0.1 but I have locally customized my server and i’d like to keep those changes*. We have several local elements to take into account: * filesystem changes (files added, removed or deleted). * configuration changes. The basic idea is to diff the existing instance with a pure new installation of the same version then apply those changes to a new provisioned version instance for staging. We can keep it at the basic filesystem approach with some simple merge strategy (theirs, ours). We can use the plugin to go into more details. For exemple using the model diff between standalone Wildfly instances we can create a cli script to reconfigure the upgraded instance in a post-installation step. <https://github.com/ehsavoie/pm/blob/upgrade-feature-pack/docs/guide/wildf... the filesystem The idea is to compare the instance to be upgraded with one instance provisioned with the same feature packs as the one we want to upgrade. The plugin will provide a list of files or regexp to be excluded. Each file will be hashed and we compare the hash + the relative path to deduce deleted, modified or added files. For textual files we can provide a diff (and the means to apply it), but maybe that should be for a later version as some kind of interaction with the user might be required. <https://github.com/ehsavoie/pm/blob/upgrade-feature-pack/docs/guide/wildf... standalone plugin This is a specialization of the upgrading algorithm: * Filtering out some of the 'unimportant' files (tmp, logs). * Creating diff of textual files (for example the realm properties) which will be applied (merging strategy à la git). * Using an embedded standalone it creates a jboss-cli script to reconfigure the server (adding/removing extensions and reconfiguring subsystems). * Deleting files that were removed. This is done on a staging upgraded instance before being copied over the old instance. I have added a diff/sync operation in standalone that is quite similar to what happens when a slave HC connects to the DC. Thus I start the current installation, and connect to it from an embedded server using the initial configuration and diff the models. This is 'experimental' but it works nicely (I was able to 'upgrade' from the standalone.xml of wildfly-core to the standalone-full.xml of wildfly). I’m talking only about the model part, I leave the files to the filesystem 'diffing' but it wiill work with managed deployments are those are added by the filesystem part and then the deployment reference is added in the model. For a future version of the tooling/plugin we might look for a way to interract more with the user (for example for applying the textual diffs to choose what to do per file instead of globally). Also currently the filters for excluding files are defined by the plugin but we could enrich them from the tooling also. <https://github.com/ehsavoie/pm/blob/upgrade-feature-pack/docs/guide/wildf... an update feature pack From the initial upgrade mecanism Alexey has seen the potential to create a feature pack instead of my initial format. Currently i’m able to create and installa a feature-pack that will supersede the initial installation with its own local modifications. Thus from my customized instance I can produce a feature pack that will allow me to reinstall the same instance. Maybe this can be also use to produce upgrade feature pack for patching. <https://github.com/ehsavoie/pm/blob/upgrade-feature-pack/docs/guide/wildf... domain mode Domain mode is a bit more complex, and we need to think how to manager the model changes. Those can be at the domain level or the host level and depending on the instance target we would need to get the changes from the domain.xml or/and the host.xml. I’m thinking about applying the same strategy as what was done for standalone : aka expose the sync operations to an embedded HC. _______________________________________________ wildfly-dev mailing list wildfly-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/wildfly-dev

Interesting. It brings up a discussion point I have been meaning to raise across the Wildfly and Wildfly-Swarm teams regarding tools for the step before this assembly step of feature-packs and fractions into a distributable archive. The issue I have seen is that when authoring a feature-pack or fraction, it is difficult to know how to configure the module dependencies. One is often starting with GAV dependencies from a spec, and it is difficult to know how those map onto modules in existing feature-packs for fractions. Do we have any tooling in this area to make this task not a trial and error effort? On Mon, Oct 2, 2017 at 7:34 AM, Emmanuel Hugonnet <ehugonne(a)redhat.com&gt; wrote: > Hi, > > For those who can take 7'26 of their time to look at what we can do with > the new provisioning tool to build and share custom feature packs. > > https://www.dropbox.com/s/84133sgsjef7pqs/feature_pack.mp4?dl=0 > > > > _______________________________________________ > wildfly-dev mailing list > wildfly-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/wildfly-dev > _______________________________________________ wildfly-dev mailing list wildfly-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/wildfly-dev

Correct. Starting from a pom that contains the artifact dependencies, how to translate this into an appropriate module.xml for the output fraction is a trial and error process currently. Does the information exist so that one could go from GAV and/or package names dependencies to the module that provides said dependencies from amongst a set of wildfly feature pack? On Tue, Oct 3, 2017 at 8:03 AM, Bob McWhirter <bmcwhirt(a)redhat.com&gt; wrote: > I think the issue (at least as I’ve found it) is that we either: > > a) scrub around in existing feature-packs and modules/ directory to > determine if there’s a module.xml matching the thing and version we need or > > b) hand-author a module.xml based upon pom.xml, adding a chance to mess > things up. > > -Bob > > On Tue, Oct 3, 2017 at 10:37 AM, Alexey Loubyansky < > alexey.loubyansky(a)redhat.com&gt; wrote: > >> It's not exactly clear to me what issue you are describing. But I can >> provide some basic info on how feature-packs are authored for the wildfly >> releases (core, servlet, full). Perhaps then you could ask a more specific >> question. >> >> A feature-pack represents a specific release. So there will be a >> feature-pack for the core, another one for the servlet distribution and >> another one for the full one. In feature-packs, modules are organized into >> packages (which is an atomic unit of content possibly with dependencies on >> other packages from the same or another feature-pack). >> When a feature-pack is generated, the packages are generated from the >> modules. Each module becomes are package in a feature-pack. And module >> dependencies become package dependencies. Is it any close to the issue you >> described? >> >> On Tue, Oct 3, 2017 at 1:51 AM, Scott Stark <sstark(a)redhat.com&gt; wrote: >> >>> Interesting. >>> >>> It brings up a discussion point I have been meaning to raise across the >>> Wildfly and Wildfly-Swarm teams regarding tools for the step before this >>> assembly step of feature-packs and fractions into a distributable archive. >>> >>> The issue I have seen is that when authoring a feature-pack or >>> fraction, it is difficult to know how to configure the module dependencies. >>> One is often starting with GAV dependencies from a spec, and it is >>> difficult to know how those map onto modules in existing feature-packs for >>> fractions. Do we have any tooling in this area to make this task not a >>> trial and error effort? >>> >>> >>> >>> On Mon, Oct 2, 2017 at 7:34 AM, Emmanuel Hugonnet <ehugonne(a)redhat.com&gt; >>> wrote: >>> >>>> Hi, >>>> >>>> For those who can take 7'26 of their time to look at what we can do >>>> with the new provisioning tool to build and share custom feature packs. >>>> >>>> https://www.dropbox.com/s/84133sgsjef7pqs/feature_pack.mp4?dl=0 >>>> >>>> >>>> >>>> _______________________________________________ >>>> wildfly-dev mailing list >>>> wildfly-dev(a)lists.jboss.org >>>> https://lists.jboss.org/mailman/listinfo/wildfly-dev >>>> >>> >>> >>> _______________________________________________ >>> wildfly-dev mailing list >>> wildfly-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/wildfly-dev >>> >> >> >

In the case of the provisioning tools, we don't generate module.xml files. Modules are provided by the developers/build process. We provide the mechanism to generate packages from the modules and then the feature-pack. When a feature-pack is generated, we know which other feature-packs it depends on. Then when we generate a package from a module we process the module dependencies and figure out on which package(s) from which feature-pack(s) the package we generate should depend on. I am still not sure whether that answers your question since it's formulated in the context of Swarm and I have a trouble translating it into the context of the feature-pack-based provisioning. I am interested in covering your requirements though. So if it makes sense to you let's continue the discussion or schedule a call to clear things out. On Wed, Oct 4, 2017 at 7:23 PM, Scott Stark <sstark(a)redhat.com <mailto:sstark@redhat.com>> wrote: Correct. Starting from a pom that contains the artifact dependencies, how to translate this into an appropriate module.xml for the output fraction is a trial and error process currently. Does the information exist so that one could go from GAV and/or package names dependencies to the module that provides said dependencies from amongst a set of wildfly feature pack? On Tue, Oct 3, 2017 at 8:03 AM, Bob McWhirter <bmcwhirt(a)redhat.com <mailto:bmcwhirt@redhat.com>> wrote: I think the issue (at least as I’ve found it) is that we either: a) scrub around in existing feature-packs and modules/ directory to determine if there’s a module.xml matching the thing and version we need or b) hand-author a module.xml based upon pom.xml, adding a chance to mess things up. -Bob On Tue, Oct 3, 2017 at 10:37 AM, Alexey Loubyansky <alexey.loubyansky(a)redhat.com <mailto:alexey.loubyansky@redhat.com>> wrote: It's not exactly clear to me what issue you are describing. But I can provide some basic info on how feature-packs are authored for the wildfly releases (core, servlet, full). Perhaps then you could ask a more specific question. A feature-pack represents a specific release. So there will be a feature-pack for the core, another one for the servlet distribution and another one for the full one. In feature-packs, modules are organized into packages (which is an atomic unit of content possibly with dependencies on other packages from the same or another feature-pack). When a feature-pack is generated, the packages are generated from the modules. Each module becomes are package in a feature-pack. And module dependencies become package dependencies. Is it any close to the issue you described? On Tue, Oct 3, 2017 at 1:51 AM, Scott Stark <sstark(a)redhat.com <mailto:sstark@redhat.com>> wrote: Interesting. It brings up a discussion point I have been meaning to raise across the Wildfly and Wildfly-Swarm teams regarding tools for the step before this assembly step of feature-packs and fractions into a distributable archive. The issue I have seen is that when authoring a feature-pack or fraction, it is difficult to know how to configure the module dependencies. One is often starting with GAV dependencies from a spec, and it is difficult to know how those map onto modules in existing feature-packs for fractions. Do we have any tooling in this area to make this task not a trial and error effort? On Mon, Oct 2, 2017 at 7:34 AM, Emmanuel Hugonnet <ehugonne(a)redhat.com <mailto:ehugonne@redhat.com>> wrote: Hi, For those who can take 7'26 of their time to look at what we can do with the new provisioning tool to build and share custom feature packs. https://www.dropbox.com/s/84133sgsjef7pqs/feature_pack.mp4?dl=0 <https://www.dropbox.com/s/84133sgsjef7pqs/feature_pack.mp4?dl=0> _______________________________________________ wildfly-dev mailing list wildfly-dev(a)lists.jboss.org <mailto:wildfly-dev@lists.jboss.org> https://lists.jboss.org/mailman/listinfo/wildfly-dev <https://lists.jboss.org/mailman/listinfo/wildfly-dev> _______________________________________________ wildfly-dev mailing list wildfly-dev(a)lists.jboss.org <mailto:wildfly-dev@lists.jboss.org> https://lists.jboss.org/mailman/listinfo/wildfly-dev <https://lists.jboss.org/mailman/listinfo/wildfly-dev> _______________________________________________ wildfly-dev mailing list wildfly-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/wildfly-dev