Not quite. I started with an empty inventory, which is why I didn't get this. After I put stuff in inventory, restarted the agent, I got this again. But - I think this is all related to my plugin vs. resource instance classloader analysis. Notice this AOPProxy is being accessed from the ApplicationServerComponent but within the runtime discovery executor doing discovery. discovery things and server component things just don't seem to work well together right now :) java.lang.ClassCastException: org.jboss.aop.generatedproxies.AOPProxy$0 at org.rhq.plugins.jbossas5.connection.RemoteProfileServiceConnectionProvider.doConnect(RemoteProfileServiceConnectionProvider.java:98) at org.rhq.plugins.jbossas5.connection.AbstractProfileServiceConnectionProvider.connect(AbstractProfileServiceConnectionProvider.java:42) at org.rhq.plugins.jbossas5.ApplicationServerComponent.connectToProfileService(ApplicationServerComponent.java:281) at org.rhq.plugins.jbossas5.ApplicationServerComponent.getAvailability(ApplicationServerComponent.java:132) at org.rhq.core.pc.inventory.RuntimeDiscoveryExecutor.discoverForResource(RuntimeDiscoveryExecutor.java:193) at org.rhq.core.pc.inventory.RuntimeDiscoveryExecutor.runtimeDiscover(RuntimeDiscoveryExecutor.java:138) at org.rhq.core.pc.inventory.RuntimeDiscoveryExecutor.call(RuntimeDiscoveryExecutor.java:98) at org.rhq.core.pc.inventory.RuntimeDiscoveryExecutor.call(RuntimeDiscoveryExecutor.java:55) ----- Original Message ----- From: mazz(a)redhat.com To: "jopr-dev" <jopr-dev(a)lists.jboss.org&gt; Sent: Saturday, July 4, 2009 12:22:40 AM GMT -05:00 US/Canada Eastern Subject: Re: [jopr-dev] classloader stuff in branches OK, after a clean rebuild, I'm not seeing this AOP class cast exception, however, I do see a very bad problem, possibly yet another flaw in the way we/I designed/implemented this. I do not expect most (all?) on this mailing list to understand this email :) I'm documenting it mostly for myself because I think better when I type out the problem. In fact, after I typed all this up, I think it helped me come up with a possible solution - mentioned at the end. But if you are involved in or needing the new classloader stuff, try to follow... First, off...from my last email - the AOP object (in the "found" variable) has this info: System.out.println("1 " + found.getClass().getInterfaces()[1]); System.out.println("2 " + found.getClass().getInterfaces()[1].getClassLoader()); which results in: 1 interface org.jboss.profileservice.spi.ProfileService 2 PluginClassLoader@157e43[parent=PluginClassLoader@11dc088[... This AOP object is obtained in the thread that is currently invoking the server component's start() method which has a context classloader of the RESOURCE INSTANCE classloader that the inventory manager created for the component. That classloader is the RESOURCE INSTANCE classloader so it has all the jboss-as-5 jars in it. Take note of the hashcodes that identify the instances. I also inspect these variables in the debugger while in the server component start(), and the hashcodes match, which tells me they are the same classloaders - this just makes sure it was able to load a profile service class and its classloader is the same as the thread context classloader: ProfileService.class.getClassLoader()=PluginClassLoader@157e43[parent=PluginClassLoader@11dc088[... Thread.currentThread().getContextClassLoader()=PluginClassLoader@157e43[parent=PluginClassLoader@11dc088[... OK, great. But now when the plugin container continues doing its thing, it tries to discover services under that jboss-as-5 server whose component was just started and it hits here: org.rhq.plugins.jbossas5.ManagedComponentDiscoveryComponent.discoverResources() at line 57: ManagementView managementView = discoveryContext.getParentResourceComponent().getConnection() This is a DISCOVERY component - we get here through the "DiscoveryComponentProxyFactory$ComponentInvocationThread.call() line: 266" - this is the thing that invokes the discovery component within the proper PLUGIN! classloader context (NOT! RESOURCE INSTANCE classloader) In the debugger, I try to get the ProfileService interface which in this discovery component and as expected, I get "<could not resolve type: org.jboss.profileservice.spi.ProfileService>". This is correct - I'm in the context of the PLUGIN classloader which doesn't have the profile service client jars, so there is no way for it to know where this ProfileService class definition can be found. That line in the discovery component can never work because that jboss-as-5 ManagementView class won't be in the plugin classloader (since it won't have the client jars). This is because this is in a discovery component - its not associated with any resource and therefore canNOT use a RESOURCE INSTANCE classloader. Instead what it uses is the PLUGIN classloader, which is this: PluginClassLoader@e77ca4[parent=PluginClassLoader@11dc088[... Notice this plugin classloader is different (@e77ca4) than the resource instance classloader (@157e43) as it should be and expected to be. That plugin classloader does not have the jboss-as-5 client jars in it. This discovery was triggered in this thread and following this code you'll see that the plugin classloader is the one that will be assigned to the thread context as expected: Daemon Thread [InventoryManager.discovery-1] DiscoveryComponentProxyFactory$ResourceDiscoveryComponentInvocationHandler.invokeInNewThread() line: 204 DiscoveryComponentProxyFactory$ResourceDiscoveryComponentInvocationHandler.invoke() line: 193 $Proxy47.discoverResources() line: not available InventoryManager.invokeDiscoveryComponent() line: 279 InventoryManager.executeComponentDiscovery() line: 1773 RuntimeDiscoveryExecutor.discoverForResource() line: 226 RuntimeDiscoveryExecutor.runtimeDiscover() line: 138 RuntimeDiscoveryExecutor.call() line: 98 RuntimeDiscoveryExecutor.call() line: 55 ... And one final gotcha. In that discovery component, it actually doesn't fail with what you would expect. It actually fails with a ClassCastException on this class: org.rhq.plugins.jbossas5.ApplicationServerComponent !!! ManagementView managementView = discoveryContext.getParentResourceComponent().getConnection().getManagementView(); The discovery component is using the parent resource to obtain the connection to the profile service so it can perform additional discovery (i.e. probe the server to discover EJB3, datasources, etc). But that parent instance was created within the RESOURCE INSTANCE CLASSLOADER. We are in the context of the PLUGIN classloader. So, the class (even though its the same name class) is actually different. So, the flaw here is any time a parent resource component has its own classloader, the child discovery component can't ever use it (because the classloaders will be different). --- Possible solution --- can't the discovery component change its context classloader if it knows it needs to get the parent resource for its connection? In other words, this discovery component knows it needs to talk to its parent resource, and it knows it doesn't have the classes that the parent has, but it DOES know (or we can give it that knowledge if need be) what that RESOURCE INSTANCE classloader is so it can switch to it: Thread.currentThread.setContextClassLoader(discoveryContext.getParentResourceComponent().getClass().getClassLoader()). That getParentResourceComponent() is the thing throwing the exception I think, so we may have to add a new getter to the discovery context to help - getParentResourceComponentClassLoader(): Thread.currentThread.setContextClassLoader(discoveryContext.getParentResourceComponentClassLoader()); ... now it might be able to successfully call ManagementView .... --- End Possible Solution --- Anyway, that's what's going on now with this classloader stuff. This is typical in how this stuff has been going... I advance the implementation some, hit a design flaw, fix the design flaw, further advance the implementation, hit the next design flaw... rinse repeat. I'm just in interation #6 now, I believe :)

I don't think we'll be able to get the PC to automatically assign the proper classloader to discovery components because it won't be possible to use the parent component classloader and have it work with child discovery components (because those child discovery components will need additional classes potentially - and they wont' be in the parent). How's this look in the discovery context object (a new method): /** * Invokes the given discovery task within the context of the * {@link #getParentResourceComponent()} parent resource component} classloader. This is needed * when the parent component has created a connection to a managed object and it is within that * managed object where child resources are to be found and discovered. Without running discovery * tasks within that parent component's classloader, it is possible that the discovery component will * not have the connection classes available within its own classloader to execute successfully. * * @param <T> the type of object returned by the task * @param task the task to invoke in the context of the parent resource component classloader * @return the object returned by the task * @throws Exception if an exception is thrown by the task */ public <T> T invokeInParentResourceComponentClassLoaderContext(Callable<T> task) throws Exception { ClassLoader originalCL = Thread.currentThread().getContextClassLoader(); try { Thread.currentThread().setContextClassLoader(this.parentComponent.getClass().getClassLoader()); return task.call(); } finally { Thread.currentThread().setContextClassLoader(originalCL); } } ----- Original Message ----- From: "John Mazzitelli" <mazz(a)redhat.com&gt; To: "jopr-dev" <jopr-dev(a)lists.jboss.org&gt; Sent: Saturday, July 4, 2009 9:00:48 AM GMT -05:00 US/Canada Eastern Subject: Re: [jopr-dev] classloader stuff in branches A continuation of the possible solution - looks like I could have the PC assign the classloader for discovery components based on the discover component's parent resource component, thus freeing the plugin dev from having to think about it. Today, I just blindly assign the plugin classloader to it, but it looks like I need to do something similar to the classloader assignments with the resource components, I have to have the PC assign a classloader based on the parent resource component and the classloader type (instance/shared). But i'm sure it won't be straight forward :} ----- Original Message ----- From: "John Mazzitelli" <mazz(a)redhat.com&gt; To: "jopr-dev" <jopr-dev(a)lists.jboss.org&gt; Sent: Saturday, July 4, 2009 12:41:26 AM GMT -05:00 US/Canada Eastern Subject: Re: [jopr-dev] classloader stuff in branches Not quite. I started with an empty inventory, which is why I didn't get this. After I put stuff in inventory, restarted the agent, I got this again. But - I think this is all related to my plugin vs. resource instance classloader analysis. Notice this AOPProxy is being accessed from the ApplicationServerComponent but within the runtime discovery executor doing discovery. discovery things and server component things just don't seem to work well together right now :) java.lang.ClassCastException: org.jboss.aop.generatedproxies.AOPProxy$0 at org.rhq.plugins.jbossas5.connection.RemoteProfileServiceConnectionProvider.doConnect(RemoteProfileServiceConnectionProvider.java:98) at org.rhq.plugins.jbossas5.connection.AbstractProfileServiceConnectionProvider.connect(AbstractProfileServiceConnectionProvider.java:42) at org.rhq.plugins.jbossas5.ApplicationServerComponent.connectToProfileService(ApplicationServerComponent.java:281) at org.rhq.plugins.jbossas5.ApplicationServerComponent.getAvailability(ApplicationServerComponent.java:132) at org.rhq.core.pc.inventory.RuntimeDiscoveryExecutor.discoverForResource(RuntimeDiscoveryExecutor.java:193) at org.rhq.core.pc.inventory.RuntimeDiscoveryExecutor.runtimeDiscover(RuntimeDiscoveryExecutor.java:138) at org.rhq.core.pc.inventory.RuntimeDiscoveryExecutor.call(RuntimeDiscoveryExecutor.java:98) at org.rhq.core.pc.inventory.RuntimeDiscoveryExecutor.call(RuntimeDiscoveryExecutor.java:55) ----- Original Message ----- From: mazz(a)redhat.com To: "jopr-dev" <jopr-dev(a)lists.jboss.org&gt; Sent: Saturday, July 4, 2009 12:22:40 AM GMT -05:00 US/Canada Eastern Subject: Re: [jopr-dev] classloader stuff in branches OK, after a clean rebuild, I'm not seeing this AOP class cast exception, however, I do see a very bad problem, possibly yet another flaw in the way we/I designed/implemented this. I do not expect most (all?) on this mailing list to understand this email :) I'm documenting it mostly for myself because I think better when I type out the problem. In fact, after I typed all this up, I think it helped me come up with a possible solution - mentioned at the end. But if you are involved in or needing the new classloader stuff, try to follow... First, off...from my last email - the AOP object (in the "found" variable) has this info: System.out.println("1 " + found.getClass().getInterfaces()[1]); System.out.println("2 " + found.getClass().getInterfaces()[1].getClassLoader()); which results in: 1 interface org.jboss.profileservice.spi.ProfileService 2 PluginClassLoader@157e43[parent=PluginClassLoader@11dc088[... This AOP object is obtained in the thread that is currently invoking the server component's start() method which has a context classloader of the RESOURCE INSTANCE classloader that the inventory manager created for the component. That classloader is the RESOURCE INSTANCE classloader so it has all the jboss-as-5 jars in it. Take note of the hashcodes that identify the instances. I also inspect these variables in the debugger while in the server component start(), and the hashcodes match, which tells me they are the same classloaders - this just makes sure it was able to load a profile service class and its classloader is the same as the thread context classloader: ProfileService.class.getClassLoader()=PluginClassLoader@157e43[parent=PluginClassLoader@11dc088[... Thread.currentThread().getContextClassLoader()=PluginClassLoader@157e43[parent=PluginClassLoader@11dc088[... OK, great. But now when the plugin container continues doing its thing, it tries to discover services under that jboss-as-5 server whose component was just started and it hits here: org.rhq.plugins.jbossas5.ManagedComponentDiscoveryComponent.discoverResources() at line 57: ManagementView managementView = discoveryContext.getParentResourceComponent().getConnection() This is a DISCOVERY component - we get here through the "DiscoveryComponentProxyFactory$ComponentInvocationThread.call() line: 266" - this is the thing that invokes the discovery component within the proper PLUGIN! classloader context (NOT! RESOURCE INSTANCE classloader) In the debugger, I try to get the ProfileService interface which in this discovery component and as expected, I get "<could not resolve type: org.jboss.profileservice.spi.ProfileService>". This is correct - I'm in the context of the PLUGIN classloader which doesn't have the profile service client jars, so there is no way for it to know where this ProfileService class definition can be found. That line in the discovery component can never work because that jboss-as-5 ManagementView class won't be in the plugin classloader (since it won't have the client jars). This is because this is in a discovery component - its not associated with any resource and therefore canNOT use a RESOURCE INSTANCE classloader. Instead what it uses is the PLUGIN classloader, which is this: PluginClassLoader@e77ca4[parent=PluginClassLoader@11dc088[... Notice this plugin classloader is different (@e77ca4) than the resource instance classloader (@157e43) as it should be and expected to be. That plugin classloader does not have the jboss-as-5 client jars in it. This discovery was triggered in this thread and following this code you'll see that the plugin classloader is the one that will be assigned to the thread context as expected: Daemon Thread [InventoryManager.discovery-1] DiscoveryComponentProxyFactory$ResourceDiscoveryComponentInvocationHandler.invokeInNewThread() line: 204 DiscoveryComponentProxyFactory$ResourceDiscoveryComponentInvocationHandler.invoke() line: 193 $Proxy47.discoverResources() line: not available InventoryManager.invokeDiscoveryComponent() line: 279 InventoryManager.executeComponentDiscovery() line: 1773 RuntimeDiscoveryExecutor.discoverForResource() line: 226 RuntimeDiscoveryExecutor.runtimeDiscover() line: 138 RuntimeDiscoveryExecutor.call() line: 98 RuntimeDiscoveryExecutor.call() line: 55 ... And one final gotcha. In that discovery component, it actually doesn't fail with what you would expect. It actually fails with a ClassCastException on this class: org.rhq.plugins.jbossas5.ApplicationServerComponent !!! ManagementView managementView = discoveryContext.getParentResourceComponent().getConnection().getManagementView(); The discovery component is using the parent resource to obtain the connection to the profile service so it can perform additional discovery (i.e. probe the server to discover EJB3, datasources, etc). But that parent instance was created within the RESOURCE INSTANCE CLASSLOADER. We are in the context of the PLUGIN classloader. So, the class (even though its the same name class) is actually different. So, the flaw here is any time a parent resource component has its own classloader, the child discovery component can't ever use it (because the classloaders will be different). --- Possible solution --- can't the discovery component change its context classloader if it knows it needs to get the parent resource for its connection? In other words, this discovery component knows it needs to talk to its parent resource, and it knows it doesn't have the classes that the parent has, but it DOES know (or we can give it that knowledge if need be) what that RESOURCE INSTANCE classloader is so it can switch to it: Thread.currentThread.setContextClassLoader(discoveryContext.getParentResourceComponent().getClass().getClassLoader()). That getParentResourceComponent() is the thing throwing the exception I think, so we may have to add a new getter to the discovery context to help - getParentResourceComponentClassLoader(): Thread.currentThread.setContextClassLoader(discoveryContext.getParentResourceComponentClassLoader()); ... now it might be able to successfully call ManagementView .... --- End Possible Solution --- Anyway, that's what's going on now with this classloader stuff. This is typical in how this stuff has been going... I advance the implementation some, hit a design flaw, fix the design flaw, further advance the implementation, hit the next design flaw... rinse repeat. I'm just in interation #6 now, I believe :)