[jopr-dev] classloader stuff in branches

Sat Jul 4 00:41:26 EDT 2009

> OK, after a clean rebuild, I'm not seeing this AOP class cast exception

Not quite. I started with an empty inventory, which is why I didn't get this. After I put stuff in inventory, restarted the agent, I got this again. But - I think this is all related to my plugin vs. resource instance classloader analysis. Notice this AOPProxy is being accessed from the ApplicationServerComponent but within the runtime discovery executor doing discovery. discovery things and server component things just don't seem to work well together right now :)

java.lang.ClassCastException: org.jboss.aop.generatedproxies.AOPProxy$0
	at org.rhq.plugins.jbossas5.connection.RemoteProfileServiceConnectionProvider.doConnect(RemoteProfileServiceConnectionProvider.java:98)
	at org.rhq.plugins.jbossas5.connection.AbstractProfileServiceConnectionProvider.connect(AbstractProfileServiceConnectionProvider.java:42)
	at org.rhq.plugins.jbossas5.ApplicationServerComponent.connectToProfileService(ApplicationServerComponent.java:281)
	at org.rhq.plugins.jbossas5.ApplicationServerComponent.getAvailability(ApplicationServerComponent.java:132)
	at org.rhq.core.pc.inventory.RuntimeDiscoveryExecutor.discoverForResource(RuntimeDiscoveryExecutor.java:193)
	at org.rhq.core.pc.inventory.RuntimeDiscoveryExecutor.runtimeDiscover(RuntimeDiscoveryExecutor.java:138)
	at org.rhq.core.pc.inventory.RuntimeDiscoveryExecutor.call(RuntimeDiscoveryExecutor.java:98)
	at org.rhq.core.pc.inventory.RuntimeDiscoveryExecutor.call(RuntimeDiscoveryExecutor.java:55)

----- Original Message -----
From: mazz at redhat.com
To: "jopr-dev" <jopr-dev at lists.jboss.org>
Sent: Saturday, July 4, 2009 12:22:40 AM GMT -05:00 US/Canada Eastern
Subject: Re: [jopr-dev] classloader stuff in branches

OK, after a clean rebuild, I'm not seeing this AOP class cast exception, however, I do see a very bad problem, possibly yet another flaw in the way we/I designed/implemented this. I do not expect most (all?) on this mailing list to understand this email :) I'm documenting it mostly for myself because I think better when I type out the problem. In fact, after I typed all this up, I think it helped me come up with a possible solution - mentioned at the end. But if you are involved in or needing the new classloader stuff, try to follow...

First, off...from my last email - the AOP object (in the "found" variable) has this info:

System.out.println("1 " + found.getClass().getInterfaces()[1]);
System.out.println("2 " + found.getClass().getInterfaces()[1].getClassLoader());

which results in:

1 interface org.jboss.profileservice.spi.ProfileService
2 PluginClassLoader at 157e43[parent=PluginClassLoader at 11dc088[...

This AOP object is obtained in the thread that is currently invoking the server component's start() method which has a context classloader of the RESOURCE INSTANCE classloader that the inventory manager created for the component. That classloader is the RESOURCE INSTANCE classloader so it has all the jboss-as-5 jars in it. Take note of the hashcodes that identify the instances. I also inspect these variables in the debugger while in the server component start(), and the hashcodes match, which tells me they are the same classloaders - this just makes sure it was able to load a profile service class and its classloader is the same as the thread context classloader:

ProfileService.class.getClassLoader()=PluginClassLoader at 157e43[parent=PluginClassLoader at 11dc088[...
Thread.currentThread().getContextClassLoader()=PluginClassLoader at 157e43[parent=PluginClassLoader at 11dc088[...

OK, great. But now when the plugin container continues doing its thing, it tries to discover services under that jboss-as-5 server whose component was just started and it hits here:

org.rhq.plugins.jbossas5.ManagedComponentDiscoveryComponent.discoverResources()
at line 57: ManagementView managementView = discoveryContext.getParentResourceComponent().getConnection() 

This is a DISCOVERY component - we get here through the "DiscoveryComponentProxyFactory$ComponentInvocationThread.call() line: 266" - this is the thing that invokes the discovery component within the proper PLUGIN! classloader context (NOT! RESOURCE INSTANCE classloader) 

In the debugger, I try to get the ProfileService interface which in this discovery component and as expected, I get "<could not resolve type: org.jboss.profileservice.spi.ProfileService>". This is correct - I'm in the context of the PLUGIN classloader which doesn't have the profile service client jars, so there is no way for it to know where this ProfileService class definition can be found.

That line in the discovery component can never work because that jboss-as-5 ManagementView class won't be in the plugin classloader (since it won't have the client jars). This is because this is in a discovery component - its not associated with any resource and therefore canNOT use a RESOURCE INSTANCE classloader. Instead what it uses is the PLUGIN classloader, which is this:

PluginClassLoader at e77ca4[parent=PluginClassLoader at 11dc088[...

Notice this plugin classloader is different (@e77ca4) than the resource instance classloader (@157e43) as it should be and expected to be. That plugin classloader does not have the jboss-as-5 client jars in it.

This discovery was triggered in this thread and following this code you'll see that the plugin classloader is the one that will be assigned to the thread context as expected:

Daemon Thread [InventoryManager.discovery-1]
DiscoveryComponentProxyFactory$ResourceDiscoveryComponentInvocationHandler.invokeInNewThread() line: 204
DiscoveryComponentProxyFactory$ResourceDiscoveryComponentInvocationHandler.invoke() line: 193	
$Proxy47.discoverResources() line: not available	
InventoryManager.invokeDiscoveryComponent() line: 279	
InventoryManager.executeComponentDiscovery() line: 1773	
RuntimeDiscoveryExecutor.discoverForResource() line: 226	
RuntimeDiscoveryExecutor.runtimeDiscover() line: 138	
RuntimeDiscoveryExecutor.call() line: 98	
RuntimeDiscoveryExecutor.call() line: 55	
...

And one final gotcha.  In that discovery component, it actually doesn't fail with what you would expect.  It actually fails with a ClassCastException on this class: org.rhq.plugins.jbossas5.ApplicationServerComponent !!!

ManagementView managementView = discoveryContext.getParentResourceComponent().getConnection().getManagementView();

The discovery component is using the parent resource to obtain the connection to the profile service so it can perform additional discovery (i.e. probe the server to discover EJB3, datasources, etc). But that parent instance was created within the RESOURCE INSTANCE CLASSLOADER. We are in the context of the PLUGIN classloader. So, the class (even though its the same name class) is actually different. So, the flaw here is any time a parent resource component has its own classloader, the child discovery component can't ever use it (because the classloaders will be different).

--- Possible solution ---

can't the discovery component change its context classloader if it knows it needs to get the parent resource for its connection? In other words, this discovery component knows it needs to talk to its parent resource, and it knows it doesn't have the classes that the parent has, but it DOES know (or we can give it that knowledge if need be) what that RESOURCE INSTANCE classloader is so it can switch to it:

Thread.currentThread.setContextClassLoader(discoveryContext.getParentResourceComponent().getClass().getClassLoader()).

That getParentResourceComponent() is the thing throwing the exception I think, so we may have to add a new getter to the discovery context to help - getParentResourceComponentClassLoader():

Thread.currentThread.setContextClassLoader(discoveryContext.getParentResourceComponentClassLoader());
... now it might be able to successfully call ManagementView ....

--- End Possible Solution ---

Anyway, that's what's going on now with this classloader stuff. This is typical in how this stuff has been going... I advance the implementation some, hit a design flaw, fix the design flaw, further advance the implementation, hit the next design flaw... rinse repeat. I'm just in interation #6 now, I believe :)