5:47 a.m.
Hi all,
Today I was looking at a module.xml somewhere down
http://github.com/jbossas/jboss-as/blob/master/build/src/main/resources/m...
and found out that a problem I was facing was caused by a dependency in
there being out of sync with the dependencies of the pom that defines
the original module.
So I was thinking, since there is a 1-1 mapping between the poms and
modules would it make sense to automatically generate the module.xml
files from the poms rather then having them in git. I realize that there
is extra information in some cases that needs to go into the module.xml
file but that information could be added to the pom in some form [1].
Otherwise this might become a bit of a maintenance nightmare IMHO...
Or is this already happening somewhere in a way that I'm not aware of?
Cheers,
David
[1] The maven-bundle-plugin does something similar for OSGi bundles:
http://felix.apache.org/site/apache-felix-maven-bundle-plugin-bnd.html
6:23 a.m.
I've been working on a maven plugin to create extra modules for tests, so what you
describe would certainly be possible. The maven plugin/dependency api is pretty nice to
work with. Once I've got my forked surefire plugin working, I can take a look if the
general consensus is that we should go with something like this
On 17 Dec 2010, at 12:47, David Bosschaert wrote:
9:07 a.m.
New subject: Automatically generate AS7 module.xml files from maven poms?
It might be a reasonable thing to generate a descriptor from a pom as a
starting point. However, they are not really one-to-one, so the majority
case will require alterations. Various reasons include:
1. Maven doesn't really separate runtime deps from compile time deps
properly (only workable way is "provided", which is rarely used).
2. Our core modules (e.g. javax.*) are not represented in maven
3. Transitive deps in maven are usually wrong (only a "bom" approach
leads to "correct" deps)
4. While a slot could be an artifact version, that's not the best
approach (you want a slot to be a compatibility group, not a minor
version so that updates are easy)
5. Path filtering can't be represented.
Although, isn't your problem more of the AS dependencies being out of
sync with a pom's transitive dep? If so the same problem exists with the
maven BOM approach that we use. (AS redefines all deps using a BOM to
avoid broken/wrong deps)
On 12/17/10 6:23 AM, Kabir Khan wrote:
--
Jason T. Greene
JBoss, a division of Red Hat
11:01 a.m.
New subject: Automatically generate AS7 module.xml files from maven poms?
In my case it was a direct dependency in a module that existed in the
pom.xml but not in the module.xml.
I'm getting slightly nervous when a dependency or code change to a
module in some build system A (the bare component) requires me to go to
a completely different build system B (AS 7) to make sure that they're
in sync.
I don't have all the answers to your questions, but just as a parallel,
in the case of the Maven Bundle Plugin [1], it adds the definition of
exports and imports to the pom, which is clearly a parallel to the Path
filtering that you're mentioning. It's done in a plugin configuration
section. I'm pretty sure that the other mismatches that you're
mentioning could be handled in a similar manner.
Stuff still needs to be declared and specified, but at least it will be
in a single place...
Cheers,
David
On 17/12/2010 15:07, Jason T. Greene wrote:
11:17 a.m.
On 12/17/2010 05:47 AM, David Bosschaert wrote:
You are incorrect; there is not a 1:1 mapping between POMs and modules.
There is a 1:1 mapping between artifacts and resource roots however.
Furthermore the problem is much more complex that it appears on the
surface. Maven build dependencies deal with fixed versions; module
dependencies use common versions within a runtime environment
(approximating a BOM) which do not match up with the build versions.
And of course, we do not control the POMs for many of our dependencies.
I do agree this is something that should (and will) be addressed but
unfortunately it's not as easy as tacking some stuff on to the POM. I
only want this to be done in a high-quality, correct way, and I am
definitely NOT interested in tying this to maven in this way. Maven
cannot survive the upcoming Java 8 modularity "epoch" in its current
form so it shouldn't be the basis of our modularity build strategy. On
the other hand we can and should *consume* Maven to acquire artifacts,
for the time being anyway.
You have to understand that the core of this problem is that there is no
relationship (nor should there be) between how we identify and
distribute modules within a specific environment (e.g. AS, but in the
future perhaps OS-level environments as well) and how Maven organizes
the actual bits (by name *and* by content). Look at Maven as just a
repository for JAR files and a set of instructions for how to build each
JAR repeatably and you're most of the way there. They just do not and
cannot map to the runtime dependency environment which can change
depending on who is using the JAR, and why, and how it is identified in
that particular environment (and why). This *must* be accommodated by
another layer.
Now the work that Kabir is doing doesn't cause this problem: his plugin
is purely for *building* modular environments, specifically test
environments, given the JAR resources plus extra environment
information. So in a sense, he's using Maven as a frontend to a
simplified version of this runtime environment layer.
--
- DML
2:39 p.m.
On 17/12/2010 17:17, David M. Lloyd wrote:
This is actually quite common in OSGi as well. In some cases there is a
1-1 mapping between maven pom.xml files (typically when the module was
written as an OSGi bundle in the first place) and in other cases there
isn't such a mapping, e.g. when a bundle is created from a number of jar
files together or portions of those jar files. When there isn't a 1-1
mapping the way this is typically handled is by creating a maven module
specifically to create the bundle from its various sources. The
maven-bundle-plugin uses the Bnd tool [1] under the covers which has
quite a rich vocabulary of commands for creating a bundle from a number
of sources, computing the Export-Package/Import-Package headers etc.
This way you can create as many packagings of a number of jars files as
you want - my main point is that as much as possible of the information
is computed (based on configuration, of course) and ideally you want
this to happen in the same buildsystem that defines the components in
the first place, to make sure that the developer is reminded to update
this information as (s)he when doing any work in the area...
Cheers,
David
[1] http://www.aqute.biz/Bnd
3:51 p.m.
On 12/19/2010 02:39 PM, David Bosschaert wrote:
I actually experimented with this. It takes around 6 seconds to
repackage a module this way on my system. If we have the AS build
produce these (100+) modules, at 6 seconds apiece, that's adding 10
minutes to the build time.
If we produce these modules externally, then some poor sap still has to
go through and create these 100+ wrapper modules, and every time we do a
release on some module it has to then be repackaged - just once for now,
but when we have multiple live AS releases out there it may be multiple
times.
Therefore I don't think this approach really scales. I think that the
most efficient option in terms of human and computer resources is to
keep only the raw bits in maven, and have the AS build drive the
construction of its own environment, at least for now.
Later on I envision an environment-oriented system where you have a
single project for an environment which is versioned as a whole whenever
any of its components are updated. This project would be
dual-functional as both a run-time BOM for projects running in the
environment as well as a tool to actually construct the environment itself.
In the somewhat distant future beyond that, I think that the idea of
using a module-oriented packaging system (sort of a "yum" for Java
modules) is the last missing piece, which will allow partial
installation and automatic update of a user-side environment. It could
use the modules dependency information (as opposed to maven info) to
correctly locate dependent modules, which it would download from a
distribution (non-maven) repository which is optimized for the
dissemination of these runtime environments.
In this way, we could produce multiple runtime environment module
repositories - a "Fedora"-like one for the community AS, which would
allow people to update their local AS images (or even generalized Java
images, at some point) at will using special Java-based management
tools; and a "RHEL"-like one for EAP which is more stable and would only
contain patches for bugs and security problems and that sort of thing.
An upgrade from one "stable" version to another would simply mean moving
to a new repository "branch version" and downloading just the updated
packages that way.
The key important difference between the module repository and the
above-mentioned OS distribution mechanisms is that we would want to at
least consider a repository-wide versioning. This way rather than
dealing with ad-hoc combinations of modules, which can be difficult to
support, we could have a single local repository version which we can
correlate to a specific server repository version, so we can easily
reproduce problems by simply "warping" a local repository to that
version and running a test case in that environment.
I agree that there is some correlation between these bits of
information. But I think in most cases, the person distributing the
package into our environment(s) is not going to be the same person who
wrote the package in the first place. And I don't think Maven is up to
the task of "knowing" the difference between the runtime module
environment and the build-time environment.
Much like how Linux distributors must re-package their packages and
re-discover dependency information, I think we will continue to have to
do some amount of manual correlation, as we are today. Our advantage is
that while we do not author some of the modules we distribute, they are
mostly in Maven, complete with dependency information that can at least
be consumed by a human if not automatically by a build system.
--
- DML
2:54 a.m.
I agree, its unreasonable to derive the runtime environment
configuration from build time POMs. We do however have the "human
wiring" dilemma, which may result in missing runtime dependencies for a
module (this is what DavidB is seeing) but may also result in
inconsistent wiring. In Antwerp we talked about the option of
post-build-validation based on module capabilities/requirements. This
could detect
* missing runtime dependencies
* split package definitions
* inconsistent class space configuration (when there are multiple slots
per module)
* other missing requirements (i.e. on a specific platform)
For this to work it'd be necessary for a module to define its
capabilities/requirements. At runtime, there should be a "gate keeper"
to the modules dependency API - the Resolver. Deployments would also
define their respective set of capabilities/requirements. The resolver
could reject a deployment if the runtime does not provide the required
set of capabilities. This could happen with very detailed error
response. Currently we have various DUPs that use the modules dependency
API. IMHO these DUPs cannot be expected to produce consistent wiring
graphs for the non-trivial setups that we expect in future. Instead a
single authority should be tasked to validate that a set of modules can
be installed in an existing setup. This must happen without modifying
the existing setup.
-thomas
On 12/19/2010 10:51 PM, David M. Lloyd wrote:
--
xxxxxxxxxxxxxxxxxxxxxxxxxxxx
Thomas Diesler
JBoss OSGi Lead
JBoss, a division of Red Hat
xxxxxxxxxxxxxxxxxxxxxxxxxxxx
11:09 a.m.
On 12/21/10 2:54 AM, Thomas Diesler wrote:
No matter where you move the dependency information, there is still
going to be some kind of human factor here.
To reiterate, we must resist the urge to do fancy things that make our
(and not our users') life easier, as it costs us performance. The only
way we are going to remain as fast as we are, is if we continue to push
ourselves.
That said, I think we can do all kinds of error checking during build,
which to me makes way more sense then any kind of runtime analysis (why
continually verify what is known to be good?).
--
Jason T. Greene
JBoss, a division of Red Hat
6:05 a.m.
In the new year I can add a few use cases to the testsuite that expose
the issue.
Let's revisit what to do when we can talk about specifics.
Have a good holiday - all
-thomas
On 12/21/2010 06:09 PM, Jason T. Greene wrote:
--
xxxxxxxxxxxxxxxxxxxxxxxxxxxx
Thomas Diesler
JBoss OSGi Lead
JBoss, a division of Red Hat
xxxxxxxxxxxxxxxxxxxxxxxxxxxx
9:32 a.m.
The module.xml files are in build because build is what assembles the
runtime environment. The individual module builds are for the most part
independent of any specific runtime environment. It doesn't make sense
to push the environment-specific bits into an environment-agnostic project.
On 01/03/2011 08:49 AM, Carlo de Wolf wrote:
--
- DML
5172
days inactive
5189
days old
11 comments
6 participants
participants (6)
-
Carlo de Wolf -
David Bosschaert -
David M. Lloyd -
Jason T. Greene -
Kabir Khan -
Thomas Diesler