On Thu, Oct 21, 2021 at 5:52 AM Yeray Borges Santana <yborgess@redhat.com> wrote:
Hello Brian,

How we should deal with the standard WildFly test suite running against the WildFly Jakarta EE 9 preview server?

I understand we could have new test failures once we are using the new Jakarta namespace artifacts. If we are running in such a situation, should we skip the failed tests on the ee9 profile if we identify its root cause as the Jakarta namespace change? Should we wait until having the test suite source code also transformed?

If moving to one of these artifacts causes a regression in the WF Preview runs of the integration testsuite, IMO that initially should be treated as a flaw in the transformation work and fixed as part of it. There may be some reason we decide to deal with the problem later via a separate issue, but that shouldn't be the default choice.

WFP is already transforming these artifacts using Batavia/Eclipse Transformer and we're testing that result. What I've described here is just a different way of accomplishing the transformation. So if this work introduces a regression we should block the change, at least until we understand the problem.

Note that I've filed a bunch of JIRAs about migrating these artifacts and in the JIRA descriptions I'm pretty prescriptive about using the approach I outline here. That's because it's impractical and inefficient to have people working on different artifacts coming up with their own solutions and then asking others to review them. But the basic goal of each of the JIRAs is to get a native jakarta namespace artifact. If for some case what I outline here just doesn't work and can't be made to work with reasonable effort, it's ok to figure out a different way to meet the basic goal.

Also, you asked about "the standard WildFly test suite running against the WildFly Jakarta EE 9 preview server", which I addressed above. It's possible too that some conversions might hit failures in the new maven module's own test execution. Of course we want to resolve those, but I don't regard such a failure as a regression, as these are really new tests. And IMO we can be a bit more forgiving about ignoring failures, at least temporarily. But we should avoid doing that. It is possible you'll see failures and they are not worth the effort of addressing, given that within 6-9 months we want to convert the original java files to jakarta.*. But if things can be fixed easily it's good to fix them. For example, this was a simple tweak in ee-9/source-transform/bean-validation that addressed a failure:


So that module isn't a pure pom.xml module; it has 1 regular test source file in it. (It was needed because the javax->jakarta change is actually to the name of the file itself, not to its content, and the source file transformation doesn't handle that. Or maybe it could/should but I didn't try and figure out how to make it happen.)


Thanks, great tutorial by the way :)

Thanks!

On Thu, Oct 21, 2021 at 1:38 AM Brian Stansberry <brian.stansberry@redhat.com> wrote:


On Wed, Oct 20, 2021 at 4:04 PM Brian Stansberry <brian.stansberry@redhat.com> wrote:
Last July, I started a thread[1] here about ways to get native jakarta namespace variants of artifacts, which we need for our move to EE 10. One of the items discussed there was "5) New maven module, transform source and build". I'm posting here as a kind of status update and tutorial about that approach.

At this point there are 11 maven modules in the WildFly main source tree[2] that are producing artifacts using the approach described there, and there are PRs open for at least a couple more. I have filed or soon will file JIRAs[3] for all the remaining WF main tree's modules that produce artifacts that WildFly Preview currently is transforming when it provisions a server[4], so there will be more coming.

Overview

The high level overview of how these work is that a pom-only maven module is created that corresponds to an existing module that produces an javax artifact. The new module's pom uses the batavia maven plugin[5] (and Eclipse Transformer under the covers) to find the source code from the existing maven module, transform it, and write the transformed output to the target/ dir of the new module, specifically in the 'generated-resources', 'generated-sources', 'generated-test-resources' and 'generated-test-sources' dirs.  That generated source is then associated with the overall maven execution for the module, and thereafter gets included as part of the subsequent compile, test, package, install and deploy goals executed for the module. So, presto we have native jakarta source available that is then compiled and tested and used to package/install/deploy binary, source and javadoc jars.[6] 

The generated source does not get committed into the git repository. The git repo only has the pom.

It's a common thing for generated source to be used in an artifact build; for example it's the technique we use to generate implementation classes from the XXXLogger interfaces we have for each of our subsystems. What we're doing here just takes this concept to the max by generating 100% of the source, including test source.

Tutorial

I'm going to use a PR I sent up yesterday to illustrate how to create one of these:


Steps:

1) Decide whether the module you want to work on is ready. If you're not the component lead responsible for the module, ask that lead. To see if a module is 'ready', look at its compile time dependencies and see if it still depends on other artifacts for which a native jakarta variant is needed and doesn't exist yet. Ask here or in zulip if you are not sure! Things can change quickly, and there also may be some edge cases where you'd think you need a jakarta namespace dependency but you really don't.

If there are dependencies that are not ready yet, stop and wait until they are available. Or be prepared for your new module to not build, at which point you can save your work and wait.

2) Create a new dir under ee-9/source-transform for your new module. If the module you are transforming isn't directly under the root of the WF source, then create a matching structure under ee-9/source-transform.  For example, I created ee-9/source-transform/jpa/spi to produce a jakarta.* variant of jpa/spi.

3) Copy the pom.xml from an *existing source-transform module* into your new dir. Easiest is to use one that comes from a dir the same # of levels below source-transform as your new dir, so a couple relative paths in your new pom are correct.  For my PR I copied over


Don't start with the pom from your source input module. Unless you want to work out how to adapt it to use the source transformation. :) Granted it's not rocket science. But it's easier for code reviewers to look at these if they look like the other ones.

4) Change the artifactId in your new pom to *exactly* match the artifactId of the module you're transforming, but with "-jakarta" appended:


5) Change the 'name' tag to something appropriate, like


6) Change the 'transformer-input-dir' maven property to point to the root of the module you are transforming:


This property is used in the batavia maven plugin config in the parent source-transform dir. Each child module sets the property to point to the correct input source for that module.

This one small tweak is the only thing you need to deal with to get the source transformation set up.

7) Configure dependencies.

a) Delete the existing 'dependency' entries in your new pom, as they are for whatever random file you copied in.

b) Copy over the 'dependency' entries from the pom of the module you are transforming.

c) For any entries where your new artifact uses a dependency with a different groupId:artifactId from the one you're transforming, change the GA. For example:

EE 8 JTA


became EE 9.1 JTA


Or, in another example

an EE 8 based 'ee' subsystem module dep:


became instead a dep on the new '-jakarta' variant:


d) A nice to have is to separate from the others in the dependency listing deps where WF Preview is using a different artifact from what standard WF is using. For example see 


This separation helps code reviewers.

8) Tell your new module's parent to include it in the build:


9) The ee-9/pom.xml maintains a dependencyManagement set for all artifacts that differ in WF Preview from what is in standard WF. These can either be artifacts whose GA is not used at all in standard WF, or ones where WF Preview uses a different version.  Add your new artifact to this dependencyManagement:


10) Allow re-use of the module.xml that incorporates this artifact between standard WF and WF Preview. This is done by modifying the module.xml to add an expression that the maven-resource-plugin replaces when preparing the module.xml resource file for use in the FP build:


The added @module.jakarta.suffix@ gets replaced either with an empty string (standard WF) or "-jakarta" (WF Preview)

I forgot to say, please note the escape slash added at the beginning of the overall expression.

11) Add your new artifact as a dependency of ee-9/feature-pack/pom.xml. This is needed so the wildfly-preview Galleon feature pack can utilize your new artifact:


12) Tell the wildfly-preview feature-pack build to ignore the no-longer relevant javax artifact you're replacing.  This saves build time, compute resources and false positives in the build log that make us think your artifact hasn't been handled yet.


(Note the precise spot to put that 'exclude' may differ, a bit. Ask if this is unclear.)

13) Add an entry for your new artifact in the license.xml file for the wildfly-preview feature back. Put it in the right alphabetical spot based on its groupId and artifactId.


14) Do a build and if good, commit and send up a PR! If your input module had tests, your new module should as well and they should run. If you're curious, have a look in the new target/generated-xxx dirs to see the source..

This sounds like a lot, and I suppose it is, but other than step 7) it's all very simple stuff, mostly things you'd do any time you add a new module to the WF source tree. Granted I'm practiced at this, but it took me about half an hour to work up the PR I've been using as an example.

If you're interested in doing one of these, and the component lead responsible for the input module is agreeable, please go for it and feel free to ask for help.



[3] High level tracker issues for this work are https://issues.redhat.com/browse/WFLY-15436 and https://issues.redhat.com/browse/WFLY-15437. I'm filing separate issues for individual pieces and am linking those to one or the other of these two trackers.

[4] https://docs.google.com/spreadsheets/d/1TekfyRb2UBCLqsPQ83WTg8XOw9qwuHVSh-xk2VEdFvk/edit#gid=926013493 shows all artifacts (not just those from the WF source tree) that the WildFly Galleon plugin was transforming in builds on Oct 14. The rows that end with '-26.0.0.Beta1-SNAPSHOT.jar' are ones that are produced by WildFly main itself. I add new tabs to that document weekly to track progress on reducing the # of artifacts being transformed by the Galleon plugin.


[6] https://repository.jboss.org/org/wildfly/wildfly-mail-jakarta/25.0.0.Final/ which was produced from a maven module with nothing in it but a pom.

Best regards,

--
Brian Stansberry
Project Lead, WildFly
He/Him/His


--
Brian Stansberry
Principal Architect, Red Hat JBoss EAP
He/Him/His
_______________________________________________
wildfly-dev mailing list -- wildfly-dev@lists.jboss.org
To unsubscribe send an email to wildfly-dev-leave@lists.jboss.org
%(web_page_url)slistinfo%(cgiext)s/%(_internal_name)s


--
Brian Stansberry
Principal Architect, Red Hat JBoss EAP
He/Him/His