Folks, a quick reminder that you cannot assume a valid Module attachment in Phase.POST_MODULE or after. An OSGi deployment that cannot resolve won't have a Module attached to the DU. There is talk about aligning the deployment phase names with Bundle life cycle terminology. IMHO Phase.POST_MODULE and Phase.INSTALL are not so lucky names because they imply meaning that may not be true. For suggested improvement see https://issues.jboss.org/browse/AS7-3585 This is related to: https://issues.jboss.org/browse/AS7-5376 cheers --thomas -- xxxxxxxxxxxxxxxxxxxxxxxxxxxx Thomas Diesler JBoss OSGi Lead JBoss, a division of Red Hat xxxxxxxxxxxxxxxxxxxxxxxxxxxx _______________________________________________ jboss-as7-dev mailing list jboss-as7-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-as7-dev

> Why would the OSGI bundle not be able to resolve, is it because is waiting for another OSGI bundle to be installed? This is by virtue of the API - BundleContext.install() does not resolve the bundle. As the method name suggests, it just installs the bundle. In the hot-deployment case it is debatable whether bundle resolution and later bundle activation should be attempted or not. By design, the order of bundle deployment is not the responsibility of the user but that of the framework. For a complex graph of interdependent bundles the user cannot possibly be asked to deploy them in the "right order". Instead the API allows to INSTALL the complete set (i.e. make the metadata available to the resolver) and later activate the bundles as needed. There are other triggers for bundle resolution too (e.g. resource access) We currently do resolve/activate during DUP processing on a trial basis. For a bundle that only has dedependencies on already installed bundles the resolve/activation works fine and the services become available. I guess this is the expected hot-deploy behaviour. A bundle that cannot resolve - for various reasons, one being the user says so - we dont attempt to start the bundle either. It would still run through all remaining DUPs but does not have a module attached.

Non-OSGi deployments that use jboss-modules metadata to define their dependencies (i.e. Dependencies clause in the manifest) have that problem too, but worse. A complex system of interdependent module deployments is likely not manageable because of this ordering issue. Even if the user gets the ordering right the first time, on server restart the notion of deployment order is lost and very likely initial deployments will fail with no osgi involved. Granted that this describes a use case that is not intended to be used for user deployments.

> the classic one is deployment of JDBC drivers that have an OSGI manifest We already removed the hack that disables OSGi for this case. The JDBC driver *is* an OSGi bundle because it contains valid OSGi metadata. It gets processed as such and should work as expected. All DUP processing is identical as before except the way module dependencies are computed and how the Module service is created. The only case where an OSGi bundle gets treated as a library jar is when it is located in an EAR/lib directory. Bundles contained in EARs are otherwise processed as OSGi sub deployments.

> we should not be allowing the presence of the OSGI subsystem to provide a different experience for users that are only after EE functionality Agreed, EE deployments should not be effected - and I don't think they are. The OSGi subsystem is not activated unless #1 you do so by management op #2 you deploy a bundle #3 some component is an injection target for the system BundleContext > We remove OSGI from the default profile, and provide a standalone-osgi.xml for users that wish to use OSGI AFAICS this would remove a few services that are already lazy and a few DUPs that deal with bundle deployments. We already have the configuration for a pure OSGi runtime as you suggest. Removing the OSGi subsystem from the default profile would not solve the need for DUP authors to be aware of OSGi deployments and code for the case of unresolved bundle deployments.

> OSGI deployment that cannot be resolved pause the deployment process until such time as they can be Yes, this is very much in line with what I think how it should work. The management API should allow the user to specify whether a deployment should get resolved/activated too. As a desired side effect this could introduce life cycle for any AS7 deployment (i.e. start/stop decoupled from deploy/undeploy). I already did some work in this direction related to in "Add notion of start/stop for deployments". It builds on top of "Allow management client to associate metadata with DeploymentUnit", which is waiting to get pulled. > which means that there will always be a Module available YES ;-) cheers --thomas On 08/15/2012 07:26 AM, Stuart Douglas wrote: > Why would the OSGI bundle not be able to resolve, is it because is waiting for another OSGI bundle to be installed? And if so, wouldn't it make more sense to pause the deployment process until the bundle can be resolved? Otherwise the behaviour will be different depending on when the bundle is resolved (e.g. if a bundle is resolved late it will not have EJB's deployed, if it is resolved early it will). > > I really hate the way that OSGI takes over and prevents the module being created, I am pretty sure that the number of users that this has caused problems for is larger than the number of users that actually use OSGI (the classic one is deployment of JDBC drivers that have an OSGI manifest). > > I think we really need a solution for this for AS 7.2, because as it currently stands we are primarily an EE app server, and we should not be allowing the presence of the OSGI subsystem to provide a different experience for users that are only after EE functionality. > > To this end, I propose the following: > > - We remove OSGI from the default profile, and provide a standalone-osgi.xml for users that wish to use OSGI, this way OSGI cannot affect users that simply want EE functionality > - OSGI deployment that cannot be resolved pause the deployment process until such time as they can be, by making the POST_MODULE DeploymentUnitPhaseService passive, which means that there will always be a Module available. > > What do you think? > > Stuart > > On 15/08/2012, at 3:05 PM, Thomas Diesler <thomas.diesler(a)jboss.com&gt; wrote: > >> Folks, >> >> a quick reminder that you cannot assume a valid Module attachment in >> Phase.POST_MODULE or after. >> >> An OSGi deployment that cannot resolve won't have a Module attached to >> the DU. There is talk about aligning the deployment phase names with >> Bundle life cycle terminology. IMHO Phase.POST_MODULE and Phase.INSTALL >> are not so lucky names because they imply meaning that may not be true. >> For suggested improvement see https://issues.jboss.org/browse/AS7-3585 >> >> This is related to: https://issues.jboss.org/browse/AS7-5376 >> >> cheers >> --thomas >> >> -- >> xxxxxxxxxxxxxxxxxxxxxxxxxxxx >> Thomas Diesler >> JBoss OSGi Lead >> JBoss, a division of Red Hat >> xxxxxxxxxxxxxxxxxxxxxxxxxxxx >> >> _______________________________________________ >> jboss-as7-dev mailing list >> jboss-as7-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev -- xxxxxxxxxxxxxxxxxxxxxxxxxxxx Thomas Diesler JBoss OSGi Lead JBoss, a division of Red Hat xxxxxxxxxxxxxxxxxxxxxxxxxxxx

On 15/08/2012, at 6:59 PM, Thomas Diesler <thomas.diesler(a)jboss.com <mailto:thomas.diesler@jboss.com>> wrote: > > Why would the OSGI bundle not be able to resolve, is it because is > waiting for another OSGI bundle to be installed? > > This is by virtue of the API - BundleContext.install() does not > resolve the bundle. As the method name suggests, it just installs the > bundle. > > In the hot-deployment case it is debatable whether bundle resolution > and later bundle activation should be attempted or not. By design, > the order of bundle deployment is not the responsibility of the user > but that of the framework. For a complex graph of interdependent > bundles the user cannot possibly be asked to deploy them in the > "right order". Instead the API allows to INSTALL the complete set > (i.e. make the metadata available to the resolver) and later activate > the bundles as needed. There are other triggers for bundle resolution > too (e.g. resource access) > > We currently do resolve/activate during DUP processing on a trial > basis. For a bundle that only has dedependencies on already installed > bundles the resolve/activation works fine and the services become > available. I guess this is the expected hot-deploy behaviour. A > bundle that cannot resolve - for various reasons, one being the user > says so - we dont attempt to start the bundle either. It would still > run through all remaining DUPs but does not have a module attached. This sounds very non-deterministic. Just to clarify, are you saying that if the user has a complex bundle deployment with lots of inter-dependencies on startup some may be resolved and some won't, and this may change on subsequent startups depending on the order in which they start?

> > Non-OSGi deployments that use jboss-modules metadata to define their > dependencies (i.e. Dependencies clause in the manifest) have that > problem too, but worse. A complex system of interdependent module > deployments is likely not manageable because of this ordering issue. > Even if the user gets the ordering right the first time, on server > restart the notion of deployment order is lost and very likely > initial deployments will fail with no osgi involved. Granted that > this describes a use case that is not intended to be used for user > deployments. No, JBoss modules uses MSC services to resolve the dependencies. At container start all deployments are now run as part of the boot ops, so as long as all deployments are present this will always work. We do need a more specified way of saying "Don't start this deployment until another deployment is done", but this is mainly for things like EJB's, not for class loading.

> > > the classic one is deployment of JDBC drivers that have an OSGI > manifest > > We already removed the hack that disables OSGi for this case. The > JDBC driver *is* an OSGi bundle because it contains valid OSGi > metadata. It gets processed as such and should work as expected. All > DUP processing is identical as before except the way module > dependencies are computed and how the Module service is created. The > only case where an OSGi bundle gets treated as a library jar is when > it is located in an EAR/lib directory. Bundles contained in EARs are > otherwise processed as OSGi sub deployments. It sounds like because we have removed the hack JDBC drivers now will not work if they fail to resolve?

> > > we should not be allowing the presence of the OSGI subsystem to > provide a different experience for users that are only after EE > functionality > > Agreed, EE deployments should not be effected - and I don't think > they are. The OSGi subsystem is not activated unless #1 you do so by > management op #2 you deploy a bundle #3 some component is an > injection target for the system BundleContext > > > We remove OSGI from the default profile, and provide a > standalone-osgi.xml for users that wish to use OSGI > > AFAICS this would remove a few services that are already lazy and a > few DUPs that deal with bundle deployments. We already have the > configuration for a pure OSGi runtime as you suggest. Removing the > OSGi subsystem from the default profile would not solve the need for > DUP authors to be aware of OSGi deployments and code for the case of > unresolved bundle deployments. Even if we resolve the module issue I still think that it would be worth making this a separate profile. Like Jaikiran I really don't like the idea of other subsystems having to code around OSGI. Another possibility we could potentially explore is a separate deployment chain for OSGI, so these DUP's do not even run if it is an OSGI deployment.

Do we have any usage data on how many of our users actually use OSGI? The more I think about it the more I think it makes sense to leave it out of the default profile. Even though you say 'it is not active unless you deploy a bundle', the thing is that many JDBC driver have OSGI metadata, so users that simply want to setup a datasource will still have OSGI activating, which is usually not what they would want.

Stuart > > > OSGI deployment that cannot be resolved pause the deployment > process until such time as they can be > > Yes, this is very much in line with what I think how it should work. > The management API should allow the user to specify whether a > deployment should get resolved/activated too. As a desired side > effect this could introduce life cycle for any AS7 deployment (i.e. > start/stop decoupled from deploy/undeploy). I already did some work > in this direction related to in "Add notion of start/stop for > deployments <https://issues.jboss.org/browse/AS7-2777>";. It builds on > top of "Allow management client to associate metadata with > DeploymentUnit <https://issues.jboss.org/browse/AS7-3694>";, which is > waiting to get pulled <https://github.com/jbossas/jboss-as/pull/2790>;. > > > which means that there will always be a Module available > > YES ;-) > > cheers > --thomas > > On 08/15/2012 07:26 AM, Stuart Douglas wrote: >> Why would the OSGI bundle not be able to resolve, is it because is waiting for another OSGI bundle to be installed? And if so, wouldn't it make more sense to pause the deployment process until the bundle can be resolved? Otherwise the behaviour will be different depending on when the bundle is resolved (e.g. if a bundle is resolved late it will not have EJB's deployed, if it is resolved early it will). >> >> I really hate the way that OSGI takes over and prevents the module being created, I am pretty sure that the number of users that this has caused problems for is larger than the number of users that actually use OSGI (the classic one is deployment of JDBC drivers that have an OSGI manifest). >> >> I think we really need a solution for this for AS 7.2, because as it currently stands we are primarily an EE app server, and we should not be allowing the presence of the OSGI subsystem to provide a different experience for users that are only after EE functionality. >> >> To this end, I propose the following: >> >> - We remove OSGI from the default profile, and provide a standalone-osgi.xml for users that wish to use OSGI, this way OSGI cannot affect users that simply want EE functionality >> - OSGI deployment that cannot be resolved pause the deployment process until such time as they can be, by making the POST_MODULE DeploymentUnitPhaseService passive, which means that there will always be a Module available. >> >> What do you think? >> >> Stuart >> >> On 15/08/2012, at 3:05 PM, Thomas Diesler&lt;thomas.diesler(a)jboss.com&gt; wrote: >> >>> Folks, >>> >>> a quick reminder that you cannot assume a valid Module attachment in >>> Phase.POST_MODULE or after. >>> >>> An OSGi deployment that cannot resolve won't have a Module attached to >>> the DU. There is talk about aligning the deployment phase names with >>> Bundle life cycle terminology. IMHO Phase.POST_MODULE and Phase.INSTALL >>> are not so lucky names because they imply meaning that may not be true. >>> For suggested improvement seehttps://issues.jboss.org/browse/AS7-3585 >>> >>> This is related to:https://issues.jboss.org/browse/AS7-5376 >>> >>> cheers >>> --thomas >>> >>> -- >>> xxxxxxxxxxxxxxxxxxxxxxxxxxxx >>> Thomas Diesler >>> JBoss OSGi Lead >>> JBoss, a division of Red Hat >>> xxxxxxxxxxxxxxxxxxxxxxxxxxxx >>> >>> _______________________________________________ >>> jboss-as7-dev mailing list >>> jboss-as7-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev > > -- > xxxxxxxxxxxxxxxxxxxxxxxxxxxx > Thomas Diesler > JBoss OSGi Lead > JBoss, a division of Red Hat > xxxxxxxxxxxxxxxxxxxxxxxxxxxx

On 15/08/2012, at 9:32 PM, Thomas Diesler <thomas.diesler(a)jboss.com <mailto:thomas.diesler@jboss.com>> wrote: > > On 08/15/2012 11:20 AM, Stuart Douglas wrote: >> >> On 15/08/2012, at 6:59 PM, Thomas Diesler <thomas.diesler(a)jboss.com >> <mailto:thomas.diesler@jboss.com>> wrote: >> >>> > Why would the OSGI bundle not be able to resolve, is it because >>> is waiting for another OSGI bundle to be installed? >>> >>> This is by virtue of the API - BundleContext.install() does not >>> resolve the bundle. As the method name suggests, it just installs >>> the bundle. >>> >>> In the hot-deployment case it is debatable whether bundle >>> resolution and later bundle activation should be attempted or not. >>> By design, the order of bundle deployment is not the responsibility >>> of the user but that of the framework. For a complex graph of >>> interdependent bundles the user cannot possibly be asked to deploy >>> them in the "right order". Instead the API allows to INSTALL the >>> complete set (i.e. make the metadata available to the resolver) and >>> later activate the bundles as needed. There are other triggers for >>> bundle resolution too (e.g. resource access) >>> >>> We currently do resolve/activate during DUP processing on a trial >>> basis. For a bundle that only has dedependencies on already >>> installed bundles the resolve/activation works fine and the >>> services become available. I guess this is the expected hot-deploy >>> behaviour. A bundle that cannot resolve - for various reasons, one >>> being the user says so - we dont attempt to start the bundle >>> either. It would still run through all remaining DUPs but does not >>> have a module attached. >> >> This sounds very non-deterministic. Just to clarify, are you saying >> that if the user has a complex bundle deployment with lots of >> inter-dependencies on startup some may be resolved and some won't, >> and this may change on subsequent startups depending on the order in >> which they start? > Yes, this is a long outstanding issue [AS7-378 > <https://issues.jboss.org/browse/AS7-378>]. I still have no guarantee > that all bundles in a given set have been INSTALLED (in OSGi > terminology) / have completed the Phase.REGISTER phase (in AS7 > terminology) when the one bundle hits the BundleResolveProcessor. The > framework records the persistent bundle state and on restart it is a > requirement that all persistent bundles reach their respective target > state for successful framework initialization. There is a little more > detail to it and I'd be more than happy to work with you to find a > consistent solution. We can take up this topic in another osgi > specific thread if you like. That would be good. It will probably be a long running thread, but this is something that I think we really need to sort out.

>> >>> >>> Non-OSGi deployments that use jboss-modules metadata to define >>> their dependencies (i.e. Dependencies clause in the manifest) have >>> that problem too, but worse. A complex system of interdependent >>> module deployments is likely not manageable because of this >>> ordering issue. Even if the user gets the ordering right the first >>> time, on server restart the notion of deployment order is lost and >>> very likely initial deployments will fail with no osgi involved. >>> Granted that this describes a use case that is not intended to be >>> used for user deployments. >> >> No, JBoss modules uses MSC services to resolve the dependencies. At >> container start all deployments are now run as part of the boot ops, >> so as long as all deployments are present this will always work. We >> do need a more specified way of saying "Don't start this deployment >> until another deployment is done", but this is mainly for things >> like EJB's, not for class loading. > Considering use case: moduleA depends on moduleB. On restart both > deployments are processed in parallel. Even with 100 other > deployments in between it is guaranteed that moduleA wont run into > "missing service on next phase" error because the module service for > B has not been installed? If so I take back the above prediction on > restart, but still hold the unmanageable claim because ordering is > delegated to the user (i.e. he must get it right the first time). > EE deployments generally don't have 100 other deployment dependencies. If they are complicated the user can simply stop the server and dump them all into the deployments directory, or deploy them in a composite op. Either way, it is deterministic, and it will fail hard on error, it will not just proceed with reduced functionality if something is missing. There is definitely still functionality that could be added here though, such as the ability to wait for another deployment before proceeding, rather than simply failing if it is not present.

>> >>> >>> > the classic one is deployment of JDBC drivers that have an OSGI >>> manifest >>> >>> We already removed the hack that disables OSGi for this case. The >>> JDBC driver *is* an OSGi bundle because it contains valid OSGi >>> metadata. It gets processed as such and should work as expected. >>> All DUP processing is identical as before except the way module >>> dependencies are computed and how the Module service is created. >>> The only case where an OSGi bundle gets treated as a library jar is >>> when it is located in an EAR/lib directory. Bundles contained in >>> EARs are otherwise processed as OSGi sub deployments. >> >> It sounds like because we have removed the hack JDBC drivers now >> will not work if they fail to resolve? > > If they fail to resolve it would be because a requirement specified > by the JDBC driver cannot be satisfied (e.g. wrong execution > environment, missing package wire). I'd say the deployment of that > driver should fail at resolve time because it would not work anyway > because of the missing wire to a valid capability. Please don't > forget that the requirements given by author should be honoured and > satisfied if you want the driver to work - they should not be ignored > or replaced by some made up hard wires that happen to work. In this > respect a JDBC driver is no different to any other OSGi bundle. > Yes, but in most cases the end user does not expect any JDBC/OSGI interaction. As far as they are concerned they are just deploying a database driver, it would not even occur to them that it has OSGI implications.

>> >>> >>> > we should not be allowing the presence of the OSGI subsystem to >>> provide a different experience for users that are only after EE >>> functionality >>> >>> Agreed, EE deployments should not be effected - and I don't think >>> they are. The OSGi subsystem is not activated unless #1 you do so >>> by management op #2 you deploy a bundle #3 some component is an >>> injection target for the system BundleContext >>> >>> > We remove OSGI from the default profile, and provide a >>> standalone-osgi.xml for users that wish to use OSGI >>> >>> AFAICS this would remove a few services that are already lazy and a >>> few DUPs that deal with bundle deployments. We already have the >>> configuration for a pure OSGi runtime as you suggest. Removing the >>> OSGi subsystem from the default profile would not solve the need >>> for DUP authors to be aware of OSGi deployments and code for the >>> case of unresolved bundle deployments. >> >> Even if we resolve the module issue I still think that it would be >> worth making this a separate profile. Like Jaikiran I really don't >> like the idea of other subsystems having to code around OSGI. >> Another possibility we could potentially explore is a separate >> deployment chain for OSGI, so these DUP's do not even run if it is >> an OSGI deployment. > The purpose of OSGi integration in AS7 is to make middleware services > that come with AS7 available to modular applications that use the > OSGi standard and vice versa (i.e. make OSGi services available to EE > components). We are not trying to build a standalone OSGi runtime and > compete with Virgo <http://www.eclipse.org/virgo/>;, Karaf > <http://karaf.apache.org/>;, etc. Instead, we are competing against > WebSphere, WebLogic, Glassfish - which AFAIK all use OSGi as their > bottom most layer and increasingly so make this tech available to > user deployments. From the business perspective the ability to > architect non-trivial modular applications in a standard way is a > requirement on the product sheet. If you are just worried about the product sheet then having this available as a separate profile does not really make any difference. It is still something that we are capable of, its just not in the default profile (like JMS and CORBA for instance).

>> >> Do we have any usage data on how many of our users actually use >> OSGI? The more I think about it the more I think it makes sense to >> leave it out of the default profile. Even though you say 'it is not >> active unless you deploy a bundle', the thing is that many JDBC >> driver have OSGI metadata, so users that simply want to setup a >> datasource will still have OSGI activating, which is usually not >> what they would want. > I have download stats on sourceforge for the jbosgi umbrella which > are around 3000/month > <http://sourceforge.net/projects/jboss/files/JBossOSGi/stats/timeline?date...;. > I also know of a few large EAP accounts that are using this tech or > have it as a decision maker for EAP yes/no. The reason that many JDBC > drivers have OSGi metadata is because they *are* OSGi bundles and > want their requirements to be honoured in a given runtime. OSGi > subsystem startup should be quick and flawless and those driver > bundles should work seamless in AS7. They currently do AFAIK - if not > I'd be interested in the details. I just want to make sure that OSGI gives both deterministic behaviour for OSGI deployments, and does not force other subsystems to have to work around it with the if(module == null) hack.

I think that if there is never going to be a module then it needs to go through its own deployment chain, as it is just going to cause problems for every other processor in POST_MODULE onwards that expects there will be one. If there is potentially going to be a module at some point then the chain should wait for the module to be available, because to do otherwise basically invites non-determistic behaviour which is really not something we want.

Just to be clear, I have nothing against OSGI, but I really thing we need to make sure it 'plays nice' with everything else.

Another thought, if we do implement passive deployments (i.e. deployments that wait for a service to become available) this code is also needed for HA singleton deployments, and any sort of inter-deployment dependency mechanism above the class loading level that we already have, so this solution potentially has other uses.

Stuart > >> >> Stuart >> >>> >>> > OSGI deployment that cannot be resolved pause the deployment >>> process until such time as they can be >>> >>> Yes, this is very much in line with what I think how it should >>> work. The management API should allow the user to specify whether a >>> deployment should get resolved/activated too. As a desired side >>> effect this could introduce life cycle for any AS7 deployment (i.e. >>> start/stop decoupled from deploy/undeploy). I already did some work >>> in this direction related to in "Add notion of start/stop for >>> deployments <https://issues.jboss.org/browse/AS7-2777>";. It builds >>> on top of "Allow management client to associate metadata with >>> DeploymentUnit <https://issues.jboss.org/browse/AS7-3694>";, which >>> is waiting to get pulled >>> <https://github.com/jbossas/jboss-as/pull/2790>;. >>> >>> > which means that there will always be a Module available >>> >>> YES ;-) >>> >>> cheers >>> --thomas >>> >>> On 08/15/2012 07:26 AM, Stuart Douglas wrote: >>>> Why would the OSGI bundle not be able to resolve, is it because is waiting for another OSGI bundle to be installed? And if so, wouldn't it make more sense to pause the deployment process until the bundle can be resolved? Otherwise the behaviour will be different depending on when the bundle is resolved (e.g. if a bundle is resolved late it will not have EJB's deployed, if it is resolved early it will). >>>> >>>> I really hate the way that OSGI takes over and prevents the module being created, I am pretty sure that the number of users that this has caused problems for is larger than the number of users that actually use OSGI (the classic one is deployment of JDBC drivers that have an OSGI manifest). >>>> >>>> I think we really need a solution for this for AS 7.2, because as it currently stands we are primarily an EE app server, and we should not be allowing the presence of the OSGI subsystem to provide a different experience for users that are only after EE functionality. >>>> >>>> To this end, I propose the following: >>>> >>>> - We remove OSGI from the default profile, and provide a standalone-osgi.xml for users that wish to use OSGI, this way OSGI cannot affect users that simply want EE functionality >>>> - OSGI deployment that cannot be resolved pause the deployment process until such time as they can be, by making the POST_MODULE DeploymentUnitPhaseService passive, which means that there will always be a Module available. >>>> >>>> What do you think? >>>> >>>> Stuart >>>> >>>> On 15/08/2012, at 3:05 PM, Thomas Diesler&lt;thomas.diesler(a)jboss.com&gt; wrote: >>>> >>>>> Folks, >>>>> >>>>> a quick reminder that you cannot assume a valid Module attachment in >>>>> Phase.POST_MODULE or after. >>>>> >>>>> An OSGi deployment that cannot resolve won't have a Module attached to >>>>> the DU. There is talk about aligning the deployment phase names with >>>>> Bundle life cycle terminology. IMHO Phase.POST_MODULE and Phase.INSTALL >>>>> are not so lucky names because they imply meaning that may not be true. >>>>> For suggested improvement seehttps://issues.jboss.org/browse/AS7-3585 >>>>> >>>>> This is related to:https://issues.jboss.org/browse/AS7-5376 >>>>> >>>>> cheers >>>>> --thomas >>>>> >>>>> -- >>>>> xxxxxxxxxxxxxxxxxxxxxxxxxxxx >>>>> Thomas Diesler >>>>> JBoss OSGi Lead >>>>> JBoss, a division of Red Hat >>>>> xxxxxxxxxxxxxxxxxxxxxxxxxxxx >>>>> >>>>> _______________________________________________ >>>>> jboss-as7-dev mailing list >>>>> jboss-as7-dev(a)lists.jboss.org >>>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev >>> >>> -- >>> xxxxxxxxxxxxxxxxxxxxxxxxxxxx >>> Thomas Diesler >>> JBoss OSGi Lead >>> JBoss, a division of Red Hat >>> xxxxxxxxxxxxxxxxxxxxxxxxxxxx >> > > -- > xxxxxxxxxxxxxxxxxxxxxxxxxxxx > Thomas Diesler > JBoss OSGi Lead > JBoss, a division of Red Hat > xxxxxxxxxxxxxxxxxxxxxxxxxxxx

Regarding ... This sounds very non-deterministic. Just to clarify, are you saying that if the user has a complex bundle deployment with lots of inter-dependencies on startup some may be resolved and some won't, and this may change on subsequent startups depending on the order in which they start? With a complex set of bundle deployments the user will have to deploy them in a known order (which is a problem in itself). There is pull request #2790 waiting that will allow the management client to have control over the auto start behaviour. So a user could first install the complete set in multiple operations and later explicitly start a selected set of bundles. This would overcome the order issue on first deploy.

Once the bundles are installed and activated the framework records their respective state. On server restart these persistent bundles are deployed in an arbitrary order but there is a guarantee backed into the Framework integration layer that ensures that the first resolve attempt is made after all persistent bundles have been installed. From the resolve perspective order also matters - you might get different wiring results depending on the order you resolve the bundles. One possible approach might be to resolve the full set of persistent bundles at once, but the guarantee for an identical wiring is still weak. A better approach would be to always resolve in a known order (i.e. sort by bundle id). The still better solution would be to persist the last known wiring graph and restore that on startup. Currently, the persistent bundles are resolved in the order they hit the BundleResolveProcessor which is arbitrary AFAIK.

I have written up the complete subsystem activation process in this article. It contains the known issues and ideas for possible solutions. Perhaps we can start from there to find a more consistent solution.

cheers --thomas On 08/15/2012 01:32 PM, Thomas Diesler wrote: > > On 08/15/2012 11:20 AM, Stuart Douglas wrote: >> >> On 15/08/2012, at 6:59 PM, Thomas Diesler <thomas.diesler(a)jboss.com&gt; wrote: >> >>> > Why would the OSGI bundle not be able to resolve, is it because is waiting for another OSGI bundle to be installed? >>> >>> This is by virtue of the API - BundleContext.install() does not resolve the bundle. As the method name suggests, it just installs the bundle. >>> >>> In the hot-deployment case it is debatable whether bundle resolution and later bundle activation should be attempted or not. By design, the order of bundle deployment is not the responsibility of the user but that of the framework. For a complex graph of interdependent bundles the user cannot possibly be asked to deploy them in the "right order". Instead the API allows to INSTALL the complete set (i.e. make the metadata available to the resolver) and later activate the bundles as needed. There are other triggers for bundle resolution too (e.g. resource access) >>> >>> We currently do resolve/activate during DUP processing on a trial basis. For a bundle that only has dedependencies on already installed bundles the resolve/activation works fine and the services become available. I guess this is the expected hot-deploy behaviour. A bundle that cannot resolve - for various reasons, one being the user says so - we dont attempt to start the bundle either. It would still run through all remaining DUPs but does not have a module attached. >> >> This sounds very non-deterministic. Just to clarify, are you saying that if the user has a complex bundle deployment with lots of inter-dependencies on startup some may be resolved and some won't, and this may change on subsequent startups depending on the order in which they start? > Yes, this is a long outstanding issue [AS7-378]. I still have no guarantee that all bundles in a given set have been INSTALLED (in OSGi terminology) / have completed the Phase.REGISTER phase (in AS7 terminology) when the one bundle hits the BundleResolveProcessor. The framework records the persistent bundle state and on restart it is a requirement that all persistent bundles reach their respective target state for successful framework initialization. There is a little more detail to it and I'd be more than happy to work with you to find a consistent solution. We can take up this topic in another osgi specific thread if you like. >> >>> >>> Non-OSGi deployments that use jboss-modules metadata to define their dependencies (i.e. Dependencies clause in the manifest) have that problem too, but worse. A complex system of interdependent module deployments is likely not manageable because of this ordering issue. Even if the user gets the ordering right the first time, on server restart the notion of deployment order is lost and very likely initial deployments will fail with no osgi involved. Granted that this describes a use case that is not intended to be used for user deployments. >> >> No, JBoss modules uses MSC services to resolve the dependencies. At container start all deployments are now run as part of the boot ops, so as long as all deployments are present this will always work. We do need a more specified way of saying "Don't start this deployment until another deployment is done", but this is mainly for things like EJB's, not for class loading. > Considering use case: moduleA depends on moduleB. On restart both deployments are processed in parallel. Even with 100 other deployments in between it is guaranteed that moduleA wont run into "missing service on next phase" error because the module service for B has not been installed? If so I take back the above prediction on restart, but still hold the unmanageable claim because ordering is delegated to the user (i.e. he must get it right the first time). > >> >>> >>> > the classic one is deployment of JDBC drivers that have an OSGI manifest >>> >>> We already removed the hack that disables OSGi for this case. The JDBC driver *is* an OSGi bundle because it contains valid OSGi metadata. It gets processed as such and should work as expected. All DUP processing is identical as before except the way module dependencies are computed and how the Module service is created. The only case where an OSGi bundle gets treated as a library jar is when it is located in an EAR/lib directory. Bundles contained in EARs are otherwise processed as OSGi sub deployments. >> >> It sounds like because we have removed the hack JDBC drivers now will not work if they fail to resolve? > > If they fail to resolve it would be because a requirement specified by the JDBC driver cannot be satisfied (e.g. wrong execution environment, missing package wire). I'd say the deployment of that driver should fail at resolve time because it would not work anyway because of the missing wire to a valid capability. Please don't forget that the requirements given by author should be honoured and satisfied if you want the driver to work - they should not be ignored or replaced by some made up hard wires that happen to work. In this respect a JDBC driver is no different to any other OSGi bundle. > >> >>> >>> > we should not be allowing the presence of the OSGI subsystem to provide a different experience for users that are only after EE functionality >>> >>> Agreed, EE deployments should not be effected - and I don't think they are. The OSGi subsystem is not activated unless #1 you do so by management op #2 you deploy a bundle #3 some component is an injection target for the system BundleContext >>> >>> > We remove OSGI from the default profile, and provide a standalone-osgi.xml for users that wish to use OSGI >>> >>> AFAICS this would remove a few services that are already lazy and a few DUPs that deal with bundle deployments. We already have the configuration for a pure OSGi runtime as you suggest. Removing the OSGi subsystem from the default profile would not solve the need for DUP authors to be aware of OSGi deployments and code for the case of unresolved bundle deployments. >> >> Even if we resolve the module issue I still think that it would be worth making this a separate profile. Like Jaikiran I really don't like the idea of other subsystems having to code around OSGI. Another possibility we could potentially explore is a separate deployment chain for OSGI, so these DUP's do not even run if it is an OSGI deployment. > The purpose of OSGi integration in AS7 is to make middleware services that come with AS7 available to modular applications that use the OSGi standard and vice versa (i.e. make OSGi services available to EE components). We are not trying to build a standalone OSGi runtime and compete with Virgo, Karaf, etc. Instead, we are competing against WebSphere, WebLogic, Glassfish - which AFAIK all use OSGi as their bottom most layer and increasingly so make this tech available to user deployments. From the business perspective the ability to architect non-trivial modular applications in a standard way is a requirement on the product sheet. >> >> Do we have any usage data on how many of our users actually use OSGI? The more I think about it the more I think it makes sense to leave it out of the default profile. Even though you say 'it is not active unless you deploy a bundle', the thing is that many JDBC driver have OSGI metadata, so users that simply want to setup a datasource will still have OSGI activating, which is usually not what they would want. > I have download stats on sourceforge for the jbosgi umbrella which are around 3000/month. I also know of a few large EAP accounts that are using this tech or have it as a decision maker for EAP yes/no. The reason that many JDBC drivers have OSGi metadata is because they *are* OSGi bundles and want their requirements to be honoured in a given runtime. OSGi subsystem startup should be quick and flawless and those driver bundles should work seamless in AS7. They currently do AFAIK - if not I'd be interested in the details. >> >> Stuart >> >>> >>> > OSGI deployment that cannot be resolved pause the deployment process until such time as they can be >>> >>> Yes, this is very much in line with what I think how it should work. The management API should allow the user to specify whether a deployment should get resolved/activated too. As a desired side effect this could introduce life cycle for any AS7 deployment (i.e. start/stop decoupled from deploy/undeploy). I already did some work in this direction related to in "Add notion of start/stop for deployments". It builds on top of "Allow management client to associate metadata with DeploymentUnit", which is waiting to get pulled. >>> >>> > which means that there will always be a Module available >>> >>> YES ;-) >>> >>> cheers >>> --thomas >>> >>> On 08/15/2012 07:26 AM, Stuart Douglas wrote: >>>> Why would the OSGI bundle not be able to resolve, is it because is waiting for another OSGI bundle to be installed? And if so, wouldn't it make more sense to pause the deployment process until the bundle can be resolved? Otherwise the behaviour will be different depending on when the bundle is resolved (e.g. if a bundle is resolved late it will not have EJB's deployed, if it is resolved early it will). >>>> >>>> I really hate the way that OSGI takes over and prevents the module being created, I am pretty sure that the number of users that this has caused problems for is larger than the number of users that actually use OSGI (the classic one is deployment of JDBC drivers that have an OSGI manifest). >>>> >>>> I think we really need a solution for this for AS 7.2, because as it currently stands we are primarily an EE app server, and we should not be allowing the presence of the OSGI subsystem to provide a different experience for users that are only after EE functionality. >>>> >>>> To this end, I propose the following: >>>> >>>> - We remove OSGI from the default profile, and provide a standalone-osgi.xml for users that wish to use OSGI, this way OSGI cannot affect users that simply want EE functionality >>>> - OSGI deployment that cannot be resolved pause the deployment process until such time as they can be, by making the POST_MODULE DeploymentUnitPhaseService passive, which means that there will always be a Module available. >>>> >>>> What do you think? >>>> >>>> Stuart >>>> >>>> On 15/08/2012, at 3:05 PM, Thomas Diesler <thomas.diesler(a)jboss.com&gt; wrote: >>>> >>>>> Folks, >>>>> >>>>> a quick reminder that you cannot assume a valid Module attachment in >>>>> Phase.POST_MODULE or after. >>>>> >>>>> An OSGi deployment that cannot resolve won't have a Module attached to >>>>> the DU. There is talk about aligning the deployment phase names with >>>>> Bundle life cycle terminology. IMHO Phase.POST_MODULE and Phase.INSTALL >>>>> are not so lucky names because they imply meaning that may not be true. >>>>> For suggested improvement see https://issues.jboss.org/browse/AS7-3585 >>>>> >>>>> This is related to: https://issues.jboss.org/browse/AS7-5376 >>>>> >>>>> cheers >>>>> --thomas >>>>> >>>>> -- >>>>> xxxxxxxxxxxxxxxxxxxxxxxxxxxx >>>>> Thomas Diesler >>>>> JBoss OSGi Lead >>>>> JBoss, a division of Red Hat >>>>> xxxxxxxxxxxxxxxxxxxxxxxxxxxx >>>>> >>>>> _______________________________________________ >>>>> jboss-as7-dev mailing list >>>>> jboss-as7-dev(a)lists.jboss.org >>>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev >>> >>> -- >>> xxxxxxxxxxxxxxxxxxxxxxxxxxxx >>> Thomas Diesler >>> JBoss OSGi Lead >>> JBoss, a division of Red Hat >>> xxxxxxxxxxxxxxxxxxxxxxxxxxxx >> > > -- > xxxxxxxxxxxxxxxxxxxxxxxxxxxx > Thomas Diesler > JBoss OSGi Lead > JBoss, a division of Red Hat > xxxxxxxxxxxxxxxxxxxxxxxxxxxx -- xxxxxxxxxxxxxxxxxxxxxxxxxxxx Thomas Diesler JBoss OSGi Lead JBoss, a division of Red Hat xxxxxxxxxxxxxxxxxxxxxxxxxxxx

Hi all, Monitoring this discussion for a while now and wanted to share my experience as a software integrator from the user perspective - based on JBoss 7.1.1.FINAL and 7.1.2.EAP. Some figures: We have a quite complex system, consisting of currently 243 OSGi-bundles, 60 enterprise bundles (mostly EARs, some WARs and SARs) and about 5 JBoss-modules with heavy dependency relations between them. Especially we have EARs depending upon bundles, as well as bundles accessing enterprise functionalities.

In the meantime, after having resolved a bunch of issues , we are quite happy with the OSGi-deployment. This is true for 7.1.1, not yet for 7.1.2 (Thomas maybe you remember me reporting a bunch of bugs on that version). Deployment of OSGi-bundles works fine, all get activated correctly and all their dependencies get resolved. However the deployment of the enterprise archives gave us head-aches and currently is only possible by a workaround for us. The problem is that the enterprise archives DO need to be deployed in the correct order, or they will fail due to missing dependencies, which then gets not resolved in the end. The way we are actually doing this is to start-up with deployment attribute enabled="false" for the enterprise deployments and then handle their activation on our own during runtime after the OSGi-container finished its deployment. So to sum it up: - Am I satisfied with OSGi-bundle-deployment? Yes. - Am I satisfied with how the overall app-server starts up with all its sub-systems and deployments: Not at all.

So I encourage you to have a look into the integration of the sub-systems, so they work together in seamless and predictable a way, I think this is what an app-server user expects. One personal word regarding the short discussion about "should we take out the OSGI-container of the default profile": From my pov, this will not solve anything. From my experience major vendors (also us) going into OSGi's direction, so this is sth. that will definitely need to get supported by a recent app-server.

Best Regards, Rico Neubauer (JBoss forum screen name MrEasy) *From:*jboss-as7-dev-bounces@lists.jboss.org [mailto:jboss-as7-dev-bounces@lists.jboss.org] *On Behalf Of *Stuart Douglas *Sent:* Friday, August 17, 2012 2:16 AM *To:* Thomas Diesler *Cc:* jboss-as7-dev(a)lists.jboss.org *Subject:* Re: [jboss-as7-dev] NPE in POST_MODULE processors On 17/08/2012, at 1:42 AM, Thomas Diesler <thomas.diesler(a)jboss.com <mailto:thomas.diesler@jboss.com>> wrote: Regarding ... /This sounds very non-deterministic. Just to clarify, are you saying that if the user has a complex bundle deployment with lots of inter-dependencies on startup some may be resolved and some won't, and this may change on subsequent startups depending on the order in which they start?/ With a complex set of bundle deployments the user will have to deploy them in a known order (which is a problem in itself). There is pull request #2790 <https://github.com/jbossas/jboss-as/pull/2790> waiting that will allow the management client to have control over the auto start behaviour. So a user could first install the complete set in multiple operations and later explicitly start a selected set of bundles. This would overcome the order issue on first deploy. I really don't like this solution. I think that the best solution here is passive deployments, that don't start POST_MODULE until all their dependencies are available. In this case it does not have to be a explicit dependency on potential future bundles, but you you could have a 'resolved' service that acts as a gate, once OSGI has resolved the bundle it creates this service, which will then trigger the deployment to continue. Once the bundles are installed and activated the framework records their respective state. On server restart these persistent bundles are deployed in an arbitrary order but there is a guarantee backed into the Framework integration layer that ensures that the first resolve attempt is made after all persistent bundles have been installed. From the resolve perspective order also matters - you might get different wiring results depending on the order you resolve the bundles. One possible approach might be to resolve the full set of persistent bundles at once, but the guarantee for an identical wiring is still weak. A better approach would be to always resolve in a known order (i.e. sort by bundle id). The still better solution would be to persist the last known wiring graph and restore that on startup. Currently, the persistent bundles are resolved in the order they hit the BundleResolveProcessor which is arbitrary AFAIK. I think that this needs to be deterministic, otherwise we will end up with a situation where deploying the same thing to a domain results in different wirings for each server in the domain. Persisting the wiring does not really help in this case. IMHO any form of non-determinism is a serious bug. Stuart I have written up the complete subsystem activation process in this article <https://community.jboss.org/wiki/OSGiSubsystemActivationProcess>;. It contains the known issues and ideas for possible solutions. Perhaps we can start from there to find a more consistent solution. cheers --thomas On 08/15/2012 01:32 PM, Thomas Diesler wrote: On 08/15/2012 11:20 AM, Stuart Douglas wrote: On 15/08/2012, at 6:59 PM, Thomas Diesler <thomas.diesler(a)jboss.com <mailto:thomas.diesler@jboss.com>> wrote: > Why would the OSGI bundle not be able to resolve, is it because is waiting for another OSGI bundle to be installed? This is by virtue of the API - BundleContext.install() does not resolve the bundle. As the method name suggests, it just installs the bundle. In the hot-deployment case it is debatable whether bundle resolution and later bundle activation should be attempted or not. By design, the order of bundle deployment is not the responsibility of the user but that of the framework. For a complex graph of interdependent bundles the user cannot possibly be asked to deploy them in the "right order". Instead the API allows to INSTALL the complete set (i.e. make the metadata available to the resolver) and later activate the bundles as needed. There are other triggers for bundle resolution too (e.g. resource access) We currently do resolve/activate during DUP processing on a trial basis. For a bundle that only has dedependencies on already installed bundles the resolve/activation works fine and the services become available. I guess this is the expected hot-deploy behaviour. A bundle that cannot resolve - for various reasons, one being the user says so - we dont attempt to start the bundle either. It would still run through all remaining DUPs but does not have a module attached. This sounds very non-deterministic. Just to clarify, are you saying that if the user has a complex bundle deployment with lots of inter-dependencies on startup some may be resolved and some won't, and this may change on subsequent startups depending on the order in which they start? Yes, this is a long outstanding issue [AS7-378 <https://issues.jboss.org/browse/AS7-378>]. I still have no guarantee that all bundles in a given set have been INSTALLED (in OSGi terminology) / have completed the Phase.REGISTER phase (in AS7 terminology) when the one bundle hits the BundleResolveProcessor. The framework records the persistent bundle state and on restart it is a requirement that all persistent bundles reach their respective target state for successful framework initialization. There is a little more detail to it and I'd be more than happy to work with you to find a consistent solution. We can take up this topic in another osgi specific thread if you like. Non-OSGi deployments that use jboss-modules metadata to define their dependencies (i.e. Dependencies clause in the manifest) have that problem too, but worse. A complex system of interdependent module deployments is likely not manageable because of this ordering issue. Even if the user gets the ordering right the first time, on server restart the notion of deployment order is lost and very likely initial deployments will fail with no osgi involved. Granted that this describes a use case that is not intended to be used for user deployments. No, JBoss modules uses MSC services to resolve the dependencies. At container start all deployments are now run as part of the boot ops, so as long as all deployments are present this will always work. We do need a more specified way of saying "Don't start this deployment until another deployment is done", but this is mainly for things like EJB's, not for class loading. Considering use case: moduleA depends on moduleB. On restart both deployments are processed in parallel. Even with 100 other deployments in between it is guaranteed that moduleA wont run into "missing service on next phase" error because the module service for B has not been installed? If so I take back the above prediction on restart, but still hold the unmanageable claim because ordering is delegated to the user (i.e. he must get it right the first time). > the classic one is deployment of JDBC drivers that have an OSGI manifest We already removed the hack that disables OSGi for this case. The JDBC driver *is* an OSGi bundle because it contains valid OSGi metadata. It gets processed as such and should work as expected. All DUP processing is identical as before except the way module dependencies are computed and how the Module service is created. The only case where an OSGi bundle gets treated as a library jar is when it is located in an EAR/lib directory. Bundles contained in EARs are otherwise processed as OSGi sub deployments. It sounds like because we have removed the hack JDBC drivers now will not work if they fail to resolve? If they fail to resolve it would be because a requirement specified by the JDBC driver cannot be satisfied (e.g. wrong execution environment, missing package wire). I'd say the deployment of that driver should fail at resolve time because it would not work anyway because of the missing wire to a valid capability. Please don't forget that the requirements given by author should be honoured and satisfied if you want the driver to work - they should not be ignored or replaced by some made up hard wires that happen to work. In this respect a JDBC driver is no different to any other OSGi bundle. > we should not be allowing the presence of the OSGI subsystem to provide a different experience for users that are only after EE functionality Agreed, EE deployments should not be effected - and I don't think they are. The OSGi subsystem is not activated unless #1 you do so by management op #2 you deploy a bundle #3 some component is an injection target for the system BundleContext > We remove OSGI from the default profile, and provide a standalone-osgi.xml for users that wish to use OSGI AFAICS this would remove a few services that are already lazy and a few DUPs that deal with bundle deployments. We already have the configuration for a pure OSGi runtime as you suggest. Removing the OSGi subsystem from the default profile would not solve the need for DUP authors to be aware of OSGi deployments and code for the case of unresolved bundle deployments. Even if we resolve the module issue I still think that it would be worth making this a separate profile. Like Jaikiran I really don't like the idea of other subsystems having to code around OSGI. Another possibility we could potentially explore is a separate deployment chain for OSGI, so these DUP's do not even run if it is an OSGI deployment. The purpose of OSGi integration in AS7 is to make middleware services that come with AS7 available to modular applications that use the OSGi standard and vice versa (i.e. make OSGi services available to EE components). We are not trying to build a standalone OSGi runtime and compete with Virgo <http://www.eclipse.org/virgo/>;, Karaf <http://karaf.apache.org/>;, etc. Instead, we are competing against WebSphere, WebLogic, Glassfish - which AFAIK all use OSGi as their bottom most layer and increasingly so make this tech available to user deployments. From the business perspective the ability to architect non-trivial modular applications in a standard way is a requirement on the product sheet. Do we have any usage data on how many of our users actually use OSGI? The more I think about it the more I think it makes sense to leave it out of the default profile. Even though you say 'it is not active unless you deploy a bundle', the thing is that many JDBC driver have OSGI metadata, so users that simply want to setup a datasource will still have OSGI activating, which is usually not what they would want. I have download stats on sourceforge for the jbosgi umbrella which are around 3000/month <http://sourceforge.net/projects/jboss/files/JBossOSGi/stats/timeline?date...;. I also know of a few large EAP accounts that are using this tech or have it as a decision maker for EAP yes/no. The reason that many JDBC drivers have OSGi metadata is because they *are* OSGi bundles and want their requirements to be honoured in a given runtime. OSGi subsystem startup should be quick and flawless and those driver bundles should work seamless in AS7. They currently do AFAIK - if not I'd be interested in the details. Stuart > OSGI deployment that cannot be resolved pause the deployment process until such time as they can be Yes, this is very much in line with what I think how it should work. The management API should allow the user to specify whether a deployment should get resolved/activated too. As a desired side effect this could introduce life cycle for any AS7 deployment (i.e. start/stop decoupled from deploy/undeploy). I already did some work in this direction related to in "Add notion of start/stop for deployments <https://issues.jboss.org/browse/AS7-2777>";. It builds on top of "Allow management client to associate metadata with DeploymentUnit <https://issues.jboss.org/browse/AS7-3694>";, which is waiting to get pulled <https://github.com/jbossas/jboss-as/pull/2790>;. > which means that there will always be a Module available YES ;-) cheers --thomas On 08/15/2012 07:26 AM, Stuart Douglas wrote: Why would the OSGI bundle not be able to resolve, is it because is waiting for another OSGI bundle to be installed? And if so, wouldn't it make more sense to pause the deployment process until the bundle can be resolved? Otherwise the behaviour will be different depending on when the bundle is resolved (e.g. if a bundle is resolved late it will not have EJB's deployed, if it is resolved early it will). I really hate the way that OSGI takes over and prevents the module being created, I am pretty sure that the number of users that this has caused problems for is larger than the number of users that actually use OSGI (the classic one is deployment of JDBC drivers that have an OSGI manifest). I think we really need a solution for this for AS 7.2, because as it currently stands we are primarily an EE app server, and we should not be allowing the presence of the OSGI subsystem to provide a different experience for users that are only after EE functionality. To this end, I propose the following: - We remove OSGI from the default profile, and provide a standalone-osgi.xml for users that wish to use OSGI, this way OSGI cannot affect users that simply want EE functionality - OSGI deployment that cannot be resolved pause the deployment process until such time as they can be, by making the POST_MODULE DeploymentUnitPhaseService passive, which means that there will always be a Module available. What do you think? Stuart On 15/08/2012, at 3:05 PM, Thomas Diesler&lt;thomas.diesler(a)jboss.com&gt; <mailto:thomas.diesler@jboss.com> wrote: Folks, a quick reminder that you cannot assume a valid Module attachment in Phase.POST_MODULE or after. An OSGi deployment that cannot resolve won't have a Module attached to the DU. There is talk about aligning the deployment phase names with Bundle life cycle terminology. IMHO Phase.POST_MODULE and Phase.INSTALL are not so lucky names because they imply meaning that may not be true. For suggested improvement seehttps://issues.jboss.org/browse/AS7-3585 This is related to:https://issues.jboss.org/browse/AS7-5376 cheers --thomas -- xxxxxxxxxxxxxxxxxxxxxxxxxxxx Thomas Diesler JBoss OSGi Lead JBoss, a division of Red Hat xxxxxxxxxxxxxxxxxxxxxxxxxxxx _______________________________________________ jboss-as7-dev mailing list jboss-as7-dev(a)lists.jboss.org <mailto:jboss-as7-dev@lists.jboss.org> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev -- xxxxxxxxxxxxxxxxxxxxxxxxxxxx Thomas Diesler JBoss OSGi Lead JBoss, a division of Red Hat xxxxxxxxxxxxxxxxxxxxxxxxxxxx -- xxxxxxxxxxxxxxxxxxxxxxxxxxxx Thomas Diesler JBoss OSGi Lead JBoss, a division of Red Hat xxxxxxxxxxxxxxxxxxxxxxxxxxxx -- xxxxxxxxxxxxxxxxxxxxxxxxxxxx Thomas Diesler JBoss OSGi Lead JBoss, a division of Red Hat xxxxxxxxxxxxxxxxxxxxxxxxxxxx ... *SEEBURGER AG* Vorstand/Seeburger Executive Board: Sitz der Gesellschaft/Registered Office: Bernd Seeburger, Axel Haas, Michael Kleeberg Edisonstr. 1 D-75015 Bretten Vorsitzender des Aufsichtsrats/Chairperson of the Seeburger Supervisory Board: Tel.: 07252 / 96 - 0 Dr. Franz Scherer Fax: 07252 / 96 - 2222 Internet: http://www.seeburger.de Registergericht/Commercial Register: e-mail: info(a)seeburger.de HRB 240708 Mannheim Dieses E-Mail ist nur für den Empfänger bestimmt, an den es gerichtet ist und kann vertrauliches bzw. unter das Berufsgeheimnis fallendes Material enthalten. Jegliche darin enthaltene Ansicht oder Meinungsäußerung ist die des Autors und stellt nicht notwendigerweise die Ansicht oder Meinung der SEEBURGER AG dar. Sind Sie nicht der Empfänger, so haben Sie diese E-Mail irrtümlich erhalten und jegliche Verwendung, Veröffentlichung, Weiterleitung, Abschrift oder jeglicher Druck dieser E-Mail ist strengstens untersagt. Weder die SEEBURGER AG noch der Absender ( Neubauer. Rico ) übernehmen die Haftung für Viren; es obliegt Ihrer Verantwortung, die E-Mail und deren Anhänge auf Viren zu prüfen. The present email addresses only the addressee which it targets and may contain confidential material that may be protected by the professional secret. The opinions reflected herein are not necessarily the one of the SEEBURGER AG. If you are not the addressee, you have accidentally got this email and are not enabled to use, publish, forward, copy or print it in any way. Neither SEEBURGER AG , nor the sender (Neubauer. Rico) are liable for viruses, being your own responsibility to check this email and its attachments for this purpose.

On 08/17/2012 02:16 AM, Stuart Douglas wrote: > > On 17/08/2012, at 1:42 AM, Thomas Diesler <thomas.diesler(a)jboss.com > <mailto:thomas.diesler@jboss.com>> wrote: > >> Regarding ... >> >> /This sounds very non-deterministic. Just to clarify, are you >> saying that if the user has a complex bundle deployment with >> lots of inter-dependencies on startup some may be resolved and >> some won't, and this may change on subsequent startups depending >> on the order in which they start?/ >> >> With a complex set of bundle deployments the user will have to >> deploy them in a known order (which is a problem in itself). There >> is pull request #2790 >> <https://github.com/jbossas/jboss-as/pull/2790> waiting that will >> allow the management client to have control over the auto start >> behaviour. So a user could first install the complete set in >> multiple operations and later explicitly start a selected set of >> bundles. This would overcome the order issue on first deploy. > > I really don't like this solution. I think that the best solution > here is passive deployments, that don't start POST_MODULE until all > their dependencies are available. In this case it does not have to be > a explicit dependency on potential future bundles, but you you could > have a 'resolved' service that acts as a gate, once OSGI has resolved > the bundle it creates this service, which will then trigger the > deployment to continue. Yes, the notion of POST_MODULE phase waiting on the Bundle.RESOLVED service (which we already have) is the right direction I would think. > >> >> Once the bundles are installed and activated the framework records >> their respective state. On server restart these persistent bundles >> are deployed in an arbitrary order but there is a guarantee backed >> into the Framework integration layer that ensures that the first >> resolve attempt is made after all persistent bundles have been >> installed. From the resolve perspective order also matters - you >> might get different wiring results depending on the order you >> resolve the bundles. One possible approach might be to resolve the >> full set of persistent bundles at once, but the guarantee for an >> identical wiring is still weak. A better approach would be to always >> resolve in a known order (i.e. sort by bundle id). The still better >> solution would be to persist the last known wiring graph and restore >> that on startup. Currently, the persistent bundles are resolved in >> the order they hit the BundleResolveProcessor which is arbitrary AFAIK. > > I think that this needs to be deterministic, otherwise we will end up > with a situation where deploying the same thing to a domain results > in different wirings for each server in the domain. Persisting the > wiring does not really help in this case. IMHO any form of > non-determinism is a serious bug. Here is how it should work IMHO #1 The user needs to have control over whether bundle deployment should be automatically resolved or not. It is perfectly ok for the user to want "just install" behaviour. It is also ok, that the user wants the bundle to resolve/start but it cannot not for one reason or another. In which case the deployment chain would wait on Bundle.RESOLVED. Its at the discretion of the framework to resolve that bundle at any time - this would normally be triggered by a class load attempt or an explicit Bundle.start() call. #2 It must be guaranteed that on restart we get the same wiring for the persistent bundles. This could be done in two ways. #2.1 The order in which the bundles hit the resolve phase must be deterministic (i.e. order of bundle id) and the resolver must guarantee to produce the same result for a given bundle set and order #2.2 Every successful resolver run records the wiring result. On restart, that wiring result is restored given that the set of persistent bundles is both present and not modified. I have a prototype of a deployment chain that waits in a certain stop phase depending on a user defined StartPolicy. There are additional start/stop management operations that make the deployment progress or reverse DUP processing respectively. Perhaps you like to have a look at https://github.com/tdiesler/jboss-as/tree/as2777 > > Stuart > >> >> I have written up the complete subsystem activation process in this >> article >> <https://community.jboss.org/wiki/OSGiSubsystemActivationProcess>;. >> It contains the known issues and ideas for possible solutions. >> Perhaps we can start from there to find a more consistent solution. >> > > > >> cheers >> --thomas >> >> On 08/15/2012 01:32 PM, Thomas Diesler wrote: >>> >>> On 08/15/2012 11:20 AM, Stuart Douglas wrote: >>>> >>>> On 15/08/2012, at 6:59 PM, Thomas Diesler >>>> <thomas.diesler(a)jboss.com <mailto:thomas.diesler@jboss.com>> wrote: >>>> >>>>> > Why would the OSGI bundle not be able to resolve, is it because >>>>> is waiting for another OSGI bundle to be installed? >>>>> >>>>> This is by virtue of the API - BundleContext.install() does not >>>>> resolve the bundle. As the method name suggests, it just installs >>>>> the bundle. >>>>> >>>>> In the hot-deployment case it is debatable whether bundle >>>>> resolution and later bundle activation should be attempted or >>>>> not. By design, the order of bundle deployment is not the >>>>> responsibility of the user but that of the framework. For a >>>>> complex graph of interdependent bundles the user cannot possibly >>>>> be asked to deploy them in the "right order". Instead the API >>>>> allows to INSTALL the complete set (i.e. make the metadata >>>>> available to the resolver) and later activate the bundles as >>>>> needed. There are other triggers for bundle resolution too (e.g. >>>>> resource access) >>>>> >>>>> We currently do resolve/activate during DUP processing on a trial >>>>> basis. For a bundle that only has dedependencies on already >>>>> installed bundles the resolve/activation works fine and the >>>>> services become available. I guess this is the expected >>>>> hot-deploy behaviour. A bundle that cannot resolve - for various >>>>> reasons, one being the user says so - we dont attempt to start >>>>> the bundle either. It would still run through all remaining DUPs >>>>> but does not have a module attached. >>>> >>>> This sounds very non-deterministic. Just to clarify, are you >>>> saying that if the user has a complex bundle deployment with lots >>>> of inter-dependencies on startup some may be resolved and some >>>> won't, and this may change on subsequent startups depending on the >>>> order in which they start? >>> Yes, this is a long outstanding issue [AS7-378 >>> <https://issues.jboss.org/browse/AS7-378>]. I still have no >>> guarantee that all bundles in a given set have been INSTALLED (in >>> OSGi terminology) / have completed the Phase.REGISTER phase (in AS7 >>> terminology) when the one bundle hits the BundleResolveProcessor. >>> The framework records the persistent bundle state and on restart it >>> is a requirement that all persistent bundles reach their respective >>> target state for successful framework initialization. There is a >>> little more detail to it and I'd be more than happy to work with >>> you to find a consistent solution. We can take up this topic in >>> another osgi specific thread if you like. >>>> >>>>> >>>>> Non-OSGi deployments that use jboss-modules metadata to define >>>>> their dependencies (i.e. Dependencies clause in the manifest) >>>>> have that problem too, but worse. A complex system of >>>>> interdependent module deployments is likely not manageable >>>>> because of this ordering issue. Even if the user gets the >>>>> ordering right the first time, on server restart the notion of >>>>> deployment order is lost and very likely initial deployments will >>>>> fail with no osgi involved. Granted that this describes a use >>>>> case that is not intended to be used for user deployments. >>>> >>>> No, JBoss modules uses MSC services to resolve the dependencies. >>>> At container start all deployments are now run as part of the boot >>>> ops, so as long as all deployments are present this will always >>>> work. We do need a more specified way of saying "Don't start this >>>> deployment until another deployment is done", but this is mainly >>>> for things like EJB's, not for class loading. >>> Considering use case: moduleA depends on moduleB. On restart both >>> deployments are processed in parallel. Even with 100 other >>> deployments in between it is guaranteed that moduleA wont run into >>> "missing service on next phase" error because the module service >>> for B has not been installed? If so I take back the above >>> prediction on restart, but still hold the unmanageable claim >>> because ordering is delegated to the user (i.e. he must get it >>> right the first time). >>> >>>> >>>>> >>>>> > the classic one is deployment of JDBC drivers that have an OSGI >>>>> manifest >>>>> >>>>> We already removed the hack that disables OSGi for this case. The >>>>> JDBC driver *is* an OSGi bundle because it contains valid OSGi >>>>> metadata. It gets processed as such and should work as expected. >>>>> All DUP processing is identical as before except the way module >>>>> dependencies are computed and how the Module service is created. >>>>> The only case where an OSGi bundle gets treated as a library jar >>>>> is when it is located in an EAR/lib directory. Bundles contained >>>>> in EARs are otherwise processed as OSGi sub deployments. >>>> >>>> It sounds like because we have removed the hack JDBC drivers now >>>> will not work if they fail to resolve? >>> >>> If they fail to resolve it would be because a requirement specified >>> by the JDBC driver cannot be satisfied (e.g. wrong execution >>> environment, missing package wire). I'd say the deployment of that >>> driver should fail at resolve time because it would not work anyway >>> because of the missing wire to a valid capability. Please don't >>> forget that the requirements given by author should be honoured and >>> satisfied if you want the driver to work - they should not be >>> ignored or replaced by some made up hard wires that happen to work. >>> In this respect a JDBC driver is no different to any other OSGi bundle. >>> >>>> >>>>> >>>>> > we should not be allowing the presence of the OSGI subsystem to >>>>> provide a different experience for users that are only after EE >>>>> functionality >>>>> >>>>> Agreed, EE deployments should not be effected - and I don't think >>>>> they are. The OSGi subsystem is not activated unless #1 you do so >>>>> by management op #2 you deploy a bundle #3 some component is an >>>>> injection target for the system BundleContext >>>>> >>>>> > We remove OSGI from the default profile, and provide a >>>>> standalone-osgi.xml for users that wish to use OSGI >>>>> >>>>> AFAICS this would remove a few services that are already lazy and >>>>> a few DUPs that deal with bundle deployments. We already have the >>>>> configuration for a pure OSGi runtime as you suggest. Removing >>>>> the OSGi subsystem from the default profile would not solve the >>>>> need for DUP authors to be aware of OSGi deployments and code for >>>>> the case of unresolved bundle deployments. >>>> >>>> Even if we resolve the module issue I still think that it would be >>>> worth making this a separate profile. Like Jaikiran I really don't >>>> like the idea of other subsystems having to code around OSGI. >>>> Another possibility we could potentially explore is a separate >>>> deployment chain for OSGI, so these DUP's do not even run if it is >>>> an OSGI deployment. >>> The purpose of OSGi integration in AS7 is to make middleware >>> services that come with AS7 available to modular applications that >>> use the OSGi standard and vice versa (i.e. make OSGi services >>> available to EE components). We are not trying to build a >>> standalone OSGi runtime and compete with Virgo >>> <http://www.eclipse.org/virgo/>;, Karaf <http://karaf.apache.org/>;, >>> etc. Instead, we are competing against WebSphere, WebLogic, >>> Glassfish - which AFAIK all use OSGi as their bottom most layer and >>> increasingly so make this tech available to user deployments. From >>> the business perspective the ability to architect non-trivial >>> modular applications in a standard way is a requirement on the >>> product sheet. >>>> >>>> Do we have any usage data on how many of our users actually use >>>> OSGI? The more I think about it the more I think it makes sense to >>>> leave it out of the default profile. Even though you say 'it is >>>> not active unless you deploy a bundle', the thing is that many >>>> JDBC driver have OSGI metadata, so users that simply want to setup >>>> a datasource will still have OSGI activating, which is usually not >>>> what they would want. >>> I have download stats on sourceforge for the jbosgi umbrella which >>> are around 3000/month >>> <http://sourceforge.net/projects/jboss/files/JBossOSGi/stats/timeline?date...;. >>> I also know of a few large EAP accounts that are using this tech or >>> have it as a decision maker for EAP yes/no. The reason that many >>> JDBC drivers have OSGi metadata is because they *are* OSGi bundles >>> and want their requirements to be honoured in a given runtime. OSGi >>> subsystem startup should be quick and flawless and those driver >>> bundles should work seamless in AS7. They currently do AFAIK - if >>> not I'd be interested in the details. >>>> >>>> Stuart >>>> >>>>> >>>>> > OSGI deployment that cannot be resolved pause the deployment >>>>> process until such time as they can be >>>>> >>>>> Yes, this is very much in line with what I think how it should >>>>> work. The management API should allow the user to specify whether >>>>> a deployment should get resolved/activated too. As a desired side >>>>> effect this could introduce life cycle for any AS7 deployment >>>>> (i.e. start/stop decoupled from deploy/undeploy). I already did >>>>> some work in this direction related to in "Add notion of >>>>> start/stop for deployments >>>>> <https://issues.jboss.org/browse/AS7-2777>";. It builds on top of >>>>> "Allow management client to associate metadata with >>>>> DeploymentUnit <https://issues.jboss.org/browse/AS7-3694>";, which >>>>> is waiting to get pulled >>>>> <https://github.com/jbossas/jboss-as/pull/2790>;. >>>>> >>>>> > which means that there will always be a Module available >>>>> >>>>> YES ;-) >>>>> >>>>> cheers >>>>> --thomas >>>>> >>>>> On 08/15/2012 07:26 AM, Stuart Douglas wrote: >>>>>> Why would the OSGI bundle not be able to resolve, is it because is waiting for another OSGI bundle to be installed? And if so, wouldn't it make more sense to pause the deployment process until the bundle can be resolved? Otherwise the behaviour will be different depending on when the bundle is resolved (e.g. if a bundle is resolved late it will not have EJB's deployed, if it is resolved early it will). >>>>>> >>>>>> I really hate the way that OSGI takes over and prevents the module being created, I am pretty sure that the number of users that this has caused problems for is larger than the number of users that actually use OSGI (the classic one is deployment of JDBC drivers that have an OSGI manifest). >>>>>> >>>>>> I think we really need a solution for this for AS 7.2, because as it currently stands we are primarily an EE app server, and we should not be allowing the presence of the OSGI subsystem to provide a different experience for users that are only after EE functionality. >>>>>> >>>>>> To this end, I propose the following: >>>>>> >>>>>> - We remove OSGI from the default profile, and provide a standalone-osgi.xml for users that wish to use OSGI, this way OSGI cannot affect users that simply want EE functionality >>>>>> - OSGI deployment that cannot be resolved pause the deployment process until such time as they can be, by making the POST_MODULE DeploymentUnitPhaseService passive, which means that there will always be a Module available. >>>>>> >>>>>> What do you think? >>>>>> >>>>>> Stuart >>>>>> >>>>>> On 15/08/2012, at 3:05 PM, Thomas Diesler&lt;thomas.diesler(a)jboss.com&gt; wrote: >>>>>> >>>>>>> Folks, >>>>>>> >>>>>>> a quick reminder that you cannot assume a valid Module attachment in >>>>>>> Phase.POST_MODULE or after. >>>>>>> >>>>>>> An OSGi deployment that cannot resolve won't have a Module attached to >>>>>>> the DU. There is talk about aligning the deployment phase names with >>>>>>> Bundle life cycle terminology. IMHO Phase.POST_MODULE and Phase.INSTALL >>>>>>> are not so lucky names because they imply meaning that may not be true. >>>>>>> For suggested improvement seehttps://issues.jboss.org/browse/AS7-3585 >>>>>>> >>>>>>> This is related to:https://issues.jboss.org/browse/AS7-5376 >>>>>>> >>>>>>> cheers >>>>>>> --thomas >>>>>>> >>>>>>> -- >>>>>>> xxxxxxxxxxxxxxxxxxxxxxxxxxxx >>>>>>> Thomas Diesler >>>>>>> JBoss OSGi Lead >>>>>>> JBoss, a division of Red Hat >>>>>>> xxxxxxxxxxxxxxxxxxxxxxxxxxxx >>>>>>> >>>>>>> _______________________________________________ >>>>>>> jboss-as7-dev mailing list >>>>>>> jboss-as7-dev(a)lists.jboss.org >>>>>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev >>>>> >>>>> -- >>>>> xxxxxxxxxxxxxxxxxxxxxxxxxxxx >>>>> Thomas Diesler >>>>> JBoss OSGi Lead >>>>> JBoss, a division of Red Hat >>>>> xxxxxxxxxxxxxxxxxxxxxxxxxxxx >>>> >>> >>> -- >>> xxxxxxxxxxxxxxxxxxxxxxxxxxxx >>> Thomas Diesler >>> JBoss OSGi Lead >>> JBoss, a division of Red Hat >>> xxxxxxxxxxxxxxxxxxxxxxxxxxxx >> >> -- >> xxxxxxxxxxxxxxxxxxxxxxxxxxxx >> Thomas Diesler >> JBoss OSGi Lead >> JBoss, a division of Red Hat >> xxxxxxxxxxxxxxxxxxxxxxxxxxxx > -- xxxxxxxxxxxxxxxxxxxxxxxxxxxx Thomas Diesler JBoss OSGi Lead JBoss, a division of Red Hat xxxxxxxxxxxxxxxxxxxxxxxxxxxx _______________________________________________ jboss-as7-dev mailing list jboss-as7-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-as7-dev

On 18/08/2012, at 2:22 PM, Thomas Diesler <thomas.diesler(a)jboss.com&gt; wrote: > Just to clarify, for non-osgi deployments on server restart we process them in parallel. For a large set, deploymentA could race many phases ahead of deploymentB with no particular order in which these deployments are processed, right? Yes, unless there are inter-module class loading dependencies, in which case a deployment will wait for the deployment it depends on to finish MODULIZE.

On 18/08/2012, at 2:22 PM, Thomas Diesler <thomas.diesler(a)jboss.com <mailto:thomas.diesler@jboss.com>> wrote: > Just to clarify, for non-osgi deployments on server restart we > process them in parallel. For a large set, deploymentA could race > many phases ahead of deploymentB with no particular order in which > these deployments are processed, right? Yes, unless there are inter-module class loading dependencies, in which case a deployment will wait for the deployment it depends on to finish MODULIZE.

> > With osgi deployments we see the resolve problem because there is a > central entity (i.e. the resolver) which works on the complete set of > metadata of the installed deployments, which implies some sort of > processing order. The minimum requirement would be that all > deployments in a given set get processed by one phase before any > deployment moves on to the next phase. I think this should be doable. When the first deployment causes the OSGI subsystem to start, it should be possible to read the model, and add a dependency on the PARSE phase of every deployment. Each OSGI deployment could then a dependency on the resolver for their PARSE phase. This should allow the resolver to start when it know that every deployment has hit the end of PARSE and no further.

> > If the initial statement is correct it is possible that other > subsystem will also have issues with the random ordering/parallel > processing of deployment sets - not only on server restart. Other subsystems will eventually be able to use a direct dependency between deployments in order to setup the dependency (i.e deployments will be able to just say in a descriptor which other deployments they depend on).

Stuart > > This would be our old friend: [AS7-378] Support notion of deployment > set <https://issues.jboss.org/browse/AS7-378> > > cheers > --thomas > > > On 08/17/2012 02:13 PM, Thomas Diesler wrote: >> >> On 08/17/2012 02:16 AM, Stuart Douglas wrote: >>> >>> On 17/08/2012, at 1:42 AM, Thomas Diesler <thomas.diesler(a)jboss.com >>> <mailto:thomas.diesler@jboss.com>> wrote: >>> >>>> Regarding ... >>>> >>>> /This sounds very non-deterministic. Just to clarify, are you >>>> saying that if the user has a complex bundle deployment with >>>> lots of inter-dependencies on startup some may be resolved and >>>> some won't, and this may change on subsequent startups >>>> depending on the order in which they start?/ >>>> >>>> With a complex set of bundle deployments the user will have to >>>> deploy them in a known order (which is a problem in itself). There >>>> is pull request #2790 >>>> <https://github.com/jbossas/jboss-as/pull/2790> waiting that will >>>> allow the management client to have control over the auto start >>>> behaviour. So a user could first install the complete set in >>>> multiple operations and later explicitly start a selected set of >>>> bundles. This would overcome the order issue on first deploy. >>> >>> I really don't like this solution. I think that the best solution >>> here is passive deployments, that don't start POST_MODULE until all >>> their dependencies are available. In this case it does not have to >>> be a explicit dependency on potential future bundles, but you you >>> could have a 'resolved' service that acts as a gate, once OSGI has >>> resolved the bundle it creates this service, which will then >>> trigger the deployment to continue. >> Yes, the notion of POST_MODULE phase waiting on the Bundle.RESOLVED >> service (which we already have) is the right direction I would think. >>> >>>> >>>> Once the bundles are installed and activated the framework records >>>> their respective state. On server restart these persistent bundles >>>> are deployed in an arbitrary order but there is a guarantee backed >>>> into the Framework integration layer that ensures that the first >>>> resolve attempt is made after all persistent bundles have been >>>> installed. From the resolve perspective order also matters - you >>>> might get different wiring results depending on the order you >>>> resolve the bundles. One possible approach might be to resolve the >>>> full set of persistent bundles at once, but the guarantee for an >>>> identical wiring is still weak. A better approach would be to >>>> always resolve in a known order (i.e. sort by bundle id). The >>>> still better solution would be to persist the last known wiring >>>> graph and restore that on startup. Currently, the persistent >>>> bundles are resolved in the order they hit the >>>> BundleResolveProcessor which is arbitrary AFAIK. >>> >>> I think that this needs to be deterministic, otherwise we will end >>> up with a situation where deploying the same thing to a domain >>> results in different wirings for each server in the domain. >>> Persisting the wiring does not really help in this case. IMHO any >>> form of non-determinism is a serious bug. >> Here is how it should work IMHO >> >> #1 The user needs to have control over whether bundle deployment >> should be automatically resolved or not. It is perfectly ok for the >> user to want "just install" behaviour. It is also ok, that the user >> wants the bundle to resolve/start but it cannot not for one reason >> or another. In which case the deployment chain would wait on >> Bundle.RESOLVED. Its at the discretion of the framework to resolve >> that bundle at any time - this would normally be triggered by a >> class load attempt or an explicit Bundle.start() call. >> >> #2 It must be guaranteed that on restart we get the same wiring for >> the persistent bundles. This could be done in two ways. #2.1 The >> order in which the bundles hit the resolve phase must be >> deterministic (i.e. order of bundle id) and the resolver must >> guarantee to produce the same result for a given bundle set and order >> #2.2 Every successful resolver run records the wiring result. On >> restart, that wiring result is restored given that the set of >> persistent bundles is both present and not modified. >> >> I have a prototype of a deployment chain that waits in a certain >> stop phase depending on a user defined StartPolicy. There are >> additional start/stop management operations that make the deployment >> progress or reverse DUP processing respectively. Perhaps you like to >> have a look at >> >> https://github.com/tdiesler/jboss-as/tree/as2777 >> >>> >>> Stuart >>> >>>> >>>> I have written up the complete subsystem activation process in >>>> this article >>>> <https://community.jboss.org/wiki/OSGiSubsystemActivationProcess>;. >>>> It contains the known issues and ideas for possible solutions. >>>> Perhaps we can start from there to find a more consistent solution. >>>> >>> >>> >>> >>>> cheers >>>> --thomas >>>> >>>> On 08/15/2012 01:32 PM, Thomas Diesler wrote: >>>>> >>>>> On 08/15/2012 11:20 AM, Stuart Douglas wrote: >>>>>> >>>>>> On 15/08/2012, at 6:59 PM, Thomas Diesler >>>>>> <thomas.diesler(a)jboss.com <mailto:thomas.diesler@jboss.com>> wrote: >>>>>> >>>>>>> > Why would the OSGI bundle not be able to resolve, is it >>>>>>> because is waiting for another OSGI bundle to be installed? >>>>>>> >>>>>>> This is by virtue of the API - BundleContext.install() does not >>>>>>> resolve the bundle. As the method name suggests, it just >>>>>>> installs the bundle. >>>>>>> >>>>>>> In the hot-deployment case it is debatable whether bundle >>>>>>> resolution and later bundle activation should be attempted or >>>>>>> not. By design, the order of bundle deployment is not the >>>>>>> responsibility of the user but that of the framework. For a >>>>>>> complex graph of interdependent bundles the user cannot >>>>>>> possibly be asked to deploy them in the "right order". Instead >>>>>>> the API allows to INSTALL the complete set (i.e. make the >>>>>>> metadata available to the resolver) and later activate the >>>>>>> bundles as needed. There are other triggers for bundle >>>>>>> resolution too (e.g. resource access) >>>>>>> >>>>>>> We currently do resolve/activate during DUP processing on a >>>>>>> trial basis. For a bundle that only has dedependencies on >>>>>>> already installed bundles the resolve/activation works fine and >>>>>>> the services become available. I guess this is the expected >>>>>>> hot-deploy behaviour. A bundle that cannot resolve - for >>>>>>> various reasons, one being the user says so - we dont attempt >>>>>>> to start the bundle either. It would still run through all >>>>>>> remaining DUPs but does not have a module attached. >>>>>> >>>>>> This sounds very non-deterministic. Just to clarify, are you >>>>>> saying that if the user has a complex bundle deployment with >>>>>> lots of inter-dependencies on startup some may be resolved and >>>>>> some won't, and this may change on subsequent startups depending >>>>>> on the order in which they start? >>>>> Yes, this is a long outstanding issue [AS7-378 >>>>> <https://issues.jboss.org/browse/AS7-378>]. I still have no >>>>> guarantee that all bundles in a given set have been INSTALLED (in >>>>> OSGi terminology) / have completed the Phase.REGISTER phase (in >>>>> AS7 terminology) when the one bundle hits the >>>>> BundleResolveProcessor. The framework records the persistent >>>>> bundle state and on restart it is a requirement that all >>>>> persistent bundles reach their respective target state for >>>>> successful framework initialization. There is a little more >>>>> detail to it and I'd be more than happy to work with you to find >>>>> a consistent solution. We can take up this topic in another osgi >>>>> specific thread if you like. >>>>>> >>>>>>> >>>>>>> Non-OSGi deployments that use jboss-modules metadata to define >>>>>>> their dependencies (i.e. Dependencies clause in the manifest) >>>>>>> have that problem too, but worse. A complex system of >>>>>>> interdependent module deployments is likely not manageable >>>>>>> because of this ordering issue. Even if the user gets the >>>>>>> ordering right the first time, on server restart the notion of >>>>>>> deployment order is lost and very likely initial deployments >>>>>>> will fail with no osgi involved. Granted that this describes a >>>>>>> use case that is not intended to be used for user deployments. >>>>>> >>>>>> No, JBoss modules uses MSC services to resolve the dependencies. >>>>>> At container start all deployments are now run as part of the >>>>>> boot ops, so as long as all deployments are present this will >>>>>> always work. We do need a more specified way of saying "Don't >>>>>> start this deployment until another deployment is done", but >>>>>> this is mainly for things like EJB's, not for class loading. >>>>> Considering use case: moduleA depends on moduleB. On restart both >>>>> deployments are processed in parallel. Even with 100 other >>>>> deployments in between it is guaranteed that moduleA wont run >>>>> into "missing service on next phase" error because the module >>>>> service for B has not been installed? If so I take back the above >>>>> prediction on restart, but still hold the unmanageable claim >>>>> because ordering is delegated to the user (i.e. he must get it >>>>> right the first time). >>>>> >>>>>> >>>>>>> >>>>>>> > the classic one is deployment of JDBC drivers that have an >>>>>>> OSGI manifest >>>>>>> >>>>>>> We already removed the hack that disables OSGi for this case. >>>>>>> The JDBC driver *is* an OSGi bundle because it contains valid >>>>>>> OSGi metadata. It gets processed as such and should work as >>>>>>> expected. All DUP processing is identical as before except the >>>>>>> way module dependencies are computed and how the Module service >>>>>>> is created. The only case where an OSGi bundle gets treated as >>>>>>> a library jar is when it is located in an EAR/lib directory. >>>>>>> Bundles contained in EARs are otherwise processed as OSGi sub >>>>>>> deployments. >>>>>> >>>>>> It sounds like because we have removed the hack JDBC drivers now >>>>>> will not work if they fail to resolve? >>>>> >>>>> If they fail to resolve it would be because a requirement >>>>> specified by the JDBC driver cannot be satisfied (e.g. wrong >>>>> execution environment, missing package wire). I'd say the >>>>> deployment of that driver should fail at resolve time because it >>>>> would not work anyway because of the missing wire to a valid >>>>> capability. Please don't forget that the requirements given by >>>>> author should be honoured and satisfied if you want the driver to >>>>> work - they should not be ignored or replaced by some made up >>>>> hard wires that happen to work. In this respect a JDBC driver is >>>>> no different to any other OSGi bundle. >>>>> >>>>>> >>>>>>> >>>>>>> > we should not be allowing the presence of the OSGI subsystem >>>>>>> to provide a different experience for users that are only after >>>>>>> EE functionality >>>>>>> >>>>>>> Agreed, EE deployments should not be effected - and I don't >>>>>>> think they are. The OSGi subsystem is not activated unless #1 >>>>>>> you do so by management op #2 you deploy a bundle #3 some >>>>>>> component is an injection target for the system BundleContext >>>>>>> >>>>>>> > We remove OSGI from the default profile, and provide a >>>>>>> standalone-osgi.xml for users that wish to use OSGI >>>>>>> >>>>>>> AFAICS this would remove a few services that are already lazy >>>>>>> and a few DUPs that deal with bundle deployments. We already >>>>>>> have the configuration for a pure OSGi runtime as you suggest. >>>>>>> Removing the OSGi subsystem from the default profile would not >>>>>>> solve the need for DUP authors to be aware of OSGi deployments >>>>>>> and code for the case of unresolved bundle deployments. >>>>>> >>>>>> Even if we resolve the module issue I still think that it would >>>>>> be worth making this a separate profile. Like Jaikiran I really >>>>>> don't like the idea of other subsystems having to code around >>>>>> OSGI. Another possibility we could potentially explore is a >>>>>> separate deployment chain for OSGI, so these DUP's do not even >>>>>> run if it is an OSGI deployment. >>>>> The purpose of OSGi integration in AS7 is to make middleware >>>>> services that come with AS7 available to modular applications >>>>> that use the OSGi standard and vice versa (i.e. make OSGi >>>>> services available to EE components). We are not trying to build >>>>> a standalone OSGi runtime and compete with Virgo >>>>> <http://www.eclipse.org/virgo/>;, Karaf >>>>> <http://karaf.apache.org/>;, etc. Instead, we are competing >>>>> against WebSphere, WebLogic, Glassfish - which AFAIK all use OSGi >>>>> as their bottom most layer and increasingly so make this tech >>>>> available to user deployments. From the business perspective the >>>>> ability to architect non-trivial modular applications in a >>>>> standard way is a requirement on the product sheet. >>>>>> >>>>>> Do we have any usage data on how many of our users actually use >>>>>> OSGI? The more I think about it the more I think it makes sense >>>>>> to leave it out of the default profile. Even though you say 'it >>>>>> is not active unless you deploy a bundle', the thing is that >>>>>> many JDBC driver have OSGI metadata, so users that simply want >>>>>> to setup a datasource will still have OSGI activating, which is >>>>>> usually not what they would want. >>>>> I have download stats on sourceforge for the jbosgi umbrella >>>>> which are around 3000/month >>>>> <http://sourceforge.net/projects/jboss/files/JBossOSGi/stats/timeline?date...;. >>>>> I also know of a few large EAP accounts that are using this tech >>>>> or have it as a decision maker for EAP yes/no. The reason that >>>>> many JDBC drivers have OSGi metadata is because they *are* OSGi >>>>> bundles and want their requirements to be honoured in a given >>>>> runtime. OSGi subsystem startup should be quick and flawless and >>>>> those driver bundles should work seamless in AS7. They currently >>>>> do AFAIK - if not I'd be interested in the details. >>>>>> >>>>>> Stuart >>>>>> >>>>>>> >>>>>>> > OSGI deployment that cannot be resolved pause the deployment >>>>>>> process until such time as they can be >>>>>>> >>>>>>> Yes, this is very much in line with what I think how it should >>>>>>> work. The management API should allow the user to specify >>>>>>> whether a deployment should get resolved/activated too. As a >>>>>>> desired side effect this could introduce life cycle for any AS7 >>>>>>> deployment (i.e. start/stop decoupled from deploy/undeploy). I >>>>>>> already did some work in this direction related to in "Add >>>>>>> notion of start/stop for deployments >>>>>>> <https://issues.jboss.org/browse/AS7-2777>";. It builds on top >>>>>>> of "Allow management client to associate metadata with >>>>>>> DeploymentUnit <https://issues.jboss.org/browse/AS7-3694>";, >>>>>>> which is waiting to get pulled >>>>>>> <https://github.com/jbossas/jboss-as/pull/2790>;. >>>>>>> >>>>>>> > which means that there will always be a Module available >>>>>>> >>>>>>> YES ;-) >>>>>>> >>>>>>> cheers >>>>>>> --thomas >>>>>>> >>>>>>> On 08/15/2012 07:26 AM, Stuart Douglas wrote: >>>>>>>> Why would the OSGI bundle not be able to resolve, is it because is waiting for another OSGI bundle to be installed? And if so, wouldn't it make more sense to pause the deployment process until the bundle can be resolved? Otherwise the behaviour will be different depending on when the bundle is resolved (e.g. if a bundle is resolved late it will not have EJB's deployed, if it is resolved early it will). >>>>>>>> >>>>>>>> I really hate the way that OSGI takes over and prevents the module being created, I am pretty sure that the number of users that this has caused problems for is larger than the number of users that actually use OSGI (the classic one is deployment of JDBC drivers that have an OSGI manifest). >>>>>>>> >>>>>>>> I think we really need a solution for this for AS 7.2, because as it currently stands we are primarily an EE app server, and we should not be allowing the presence of the OSGI subsystem to provide a different experience for users that are only after EE functionality. >>>>>>>> >>>>>>>> To this end, I propose the following: >>>>>>>> >>>>>>>> - We remove OSGI from the default profile, and provide a standalone-osgi.xml for users that wish to use OSGI, this way OSGI cannot affect users that simply want EE functionality >>>>>>>> - OSGI deployment that cannot be resolved pause the deployment process until such time as they can be, by making the POST_MODULE DeploymentUnitPhaseService passive, which means that there will always be a Module available. >>>>>>>> >>>>>>>> What do you think? >>>>>>>> >>>>>>>> Stuart >>>>>>>> >>>>>>>> On 15/08/2012, at 3:05 PM, Thomas Diesler&lt;thomas.diesler(a)jboss.com&gt; wrote: >>>>>>>> >>>>>>>>> Folks, >>>>>>>>> >>>>>>>>> a quick reminder that you cannot assume a valid Module attachment in >>>>>>>>> Phase.POST_MODULE or after. >>>>>>>>> >>>>>>>>> An OSGi deployment that cannot resolve won't have a Module attached to >>>>>>>>> the DU. There is talk about aligning the deployment phase names with >>>>>>>>> Bundle life cycle terminology. IMHO Phase.POST_MODULE and Phase.INSTALL >>>>>>>>> are not so lucky names because they imply meaning that may not be true. >>>>>>>>> For suggested improvement seehttps://issues.jboss.org/browse/AS7-3585 >>>>>>>>> >>>>>>>>> This is related to:https://issues.jboss.org/browse/AS7-5376 >>>>>>>>> >>>>>>>>> cheers >>>>>>>>> --thomas >>>>>>>>> >>>>>>>>> -- >>>>>>>>> xxxxxxxxxxxxxxxxxxxxxxxxxxxx >>>>>>>>> Thomas Diesler >>>>>>>>> JBoss OSGi Lead >>>>>>>>> JBoss, a division of Red Hat >>>>>>>>> xxxxxxxxxxxxxxxxxxxxxxxxxxxx >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> jboss-as7-dev mailing list >>>>>>>>> jboss-as7-dev(a)lists.jboss.org >>>>>>>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev >>>>>>> >>>>>>> -- >>>>>>> xxxxxxxxxxxxxxxxxxxxxxxxxxxx >>>>>>> Thomas Diesler >>>>>>> JBoss OSGi Lead >>>>>>> JBoss, a division of Red Hat >>>>>>> xxxxxxxxxxxxxxxxxxxxxxxxxxxx >>>>>> >>>>> >>>>> -- >>>>> xxxxxxxxxxxxxxxxxxxxxxxxxxxx >>>>> Thomas Diesler >>>>> JBoss OSGi Lead >>>>> JBoss, a division of Red Hat >>>>> xxxxxxxxxxxxxxxxxxxxxxxxxxxx >>>> >>>> -- >>>> xxxxxxxxxxxxxxxxxxxxxxxxxxxx >>>> Thomas Diesler >>>> JBoss OSGi Lead >>>> JBoss, a division of Red Hat >>>> xxxxxxxxxxxxxxxxxxxxxxxxxxxx >>> >> >> -- >> xxxxxxxxxxxxxxxxxxxxxxxxxxxx >> Thomas Diesler >> JBoss OSGi Lead >> JBoss, a division of Red Hat >> xxxxxxxxxxxxxxxxxxxxxxxxxxxx >> >> >> _______________________________________________ >> jboss-as7-dev mailing list >> jboss-as7-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev > > -- > xxxxxxxxxxxxxxxxxxxxxxxxxxxx > Thomas Diesler > JBoss OSGi Lead > JBoss, a division of Red Hat > xxxxxxxxxxxxxxxxxxxxxxxxxxxx

IMHO Phase.POST_MODULE and Phase.INSTALL are not so lucky names because they imply meaning that may not be true. For suggested improvement see https://issues.jboss.org/browse/AS7-3585