classloaders
by mazz＠redhat.com
I have 3 JBossAS5 instances running - two are running out of one install directory (with -c default1 and -c default2) and another is running out of a second copy (with -c default3).
Look at the attached (image isn't great, had to lower quality to shrink its size). There are 6 resource classloaders created and actively assigned to resources. You can see the three JBossAS5 classloaders (all of jboss servers that are running are identical so you see they all have 23 resources associated with them). The jars in the classloaders look all correct.
This is all as expected. Just wanted to share :)
15 years, 5 months
Re: [jopr-dev] classloader stuff in branches
by mazz＠redhat.com
OK, after a clean rebuild, I'm not seeing this AOP class cast exception, however, I do see a very bad problem, possibly yet another flaw in the way we/I designed/implemented this. I do not expect most (all?) on this mailing list to understand this email :) I'm documenting it mostly for myself because I think better when I type out the problem. In fact, after I typed all this up, I think it helped me come up with a possible solution - mentioned at the end. But if you are involved in or needing the new classloader stuff, try to follow...
First, off...from my last email - the AOP object (in the "found" variable) has this info:
System.out.println("1 " + found.getClass().getInterfaces()[1]);
System.out.println("2 " + found.getClass().getInterfaces()[1].getClassLoader());
which results in:
1 interface org.jboss.profileservice.spi.ProfileService
2 PluginClassLoader@157e43[parent=PluginClassLoader@11dc088[...
This AOP object is obtained in the thread that is currently invoking the server component's start() method which has a context classloader of the RESOURCE INSTANCE classloader that the inventory manager created for the component. That classloader is the RESOURCE INSTANCE classloader so it has all the jboss-as-5 jars in it. Take note of the hashcodes that identify the instances. I also inspect these variables in the debugger while in the server component start(), and the hashcodes match, which tells me they are the same classloaders - this just makes sure it was able to load a profile service class and its classloader is the same as the thread context classloader:
ProfileService.class.getClassLoader()=PluginClassLoader@157e43[parent=PluginClassLoader@11dc088[...
Thread.currentThread().getContextClassLoader()=PluginClassLoader@157e43[parent=PluginClassLoader@11dc088[...
OK, great. But now when the plugin container continues doing its thing, it tries to discover services under that jboss-as-5 server whose component was just started and it hits here:
org.rhq.plugins.jbossas5.ManagedComponentDiscoveryComponent.discoverResources()
at line 57: ManagementView managementView = discoveryContext.getParentResourceComponent().getConnection()
This is a DISCOVERY component - we get here through the "DiscoveryComponentProxyFactory$ComponentInvocationThread.call() line: 266" - this is the thing that invokes the discovery component within the proper PLUGIN! classloader context (NOT! RESOURCE INSTANCE classloader)
In the debugger, I try to get the ProfileService interface which in this discovery component and as expected, I get "<could not resolve type: org.jboss.profileservice.spi.ProfileService>". This is correct - I'm in the context of the PLUGIN classloader which doesn't have the profile service client jars, so there is no way for it to know where this ProfileService class definition can be found.
That line in the discovery component can never work because that jboss-as-5 ManagementView class won't be in the plugin classloader (since it won't have the client jars). This is because this is in a discovery component - its not associated with any resource and therefore canNOT use a RESOURCE INSTANCE classloader. Instead what it uses is the PLUGIN classloader, which is this:
PluginClassLoader@e77ca4[parent=PluginClassLoader@11dc088[...
Notice this plugin classloader is different (@e77ca4) than the resource instance classloader (@157e43) as it should be and expected to be. That plugin classloader does not have the jboss-as-5 client jars in it.
This discovery was triggered in this thread and following this code you'll see that the plugin classloader is the one that will be assigned to the thread context as expected:
Daemon Thread [InventoryManager.discovery-1]
DiscoveryComponentProxyFactory$ResourceDiscoveryComponentInvocationHandler.invokeInNewThread() line: 204
DiscoveryComponentProxyFactory$ResourceDiscoveryComponentInvocationHandler.invoke() line: 193
$Proxy47.discoverResources() line: not available
InventoryManager.invokeDiscoveryComponent() line: 279
InventoryManager.executeComponentDiscovery() line: 1773
RuntimeDiscoveryExecutor.discoverForResource() line: 226
RuntimeDiscoveryExecutor.runtimeDiscover() line: 138
RuntimeDiscoveryExecutor.call() line: 98
RuntimeDiscoveryExecutor.call() line: 55
...
And one final gotcha. In that discovery component, it actually doesn't fail with what you would expect. It actually fails with a ClassCastException on this class: org.rhq.plugins.jbossas5.ApplicationServerComponent !!!
ManagementView managementView = discoveryContext.getParentResourceComponent().getConnection().getManagementView();
The discovery component is using the parent resource to obtain the connection to the profile service so it can perform additional discovery (i.e. probe the server to discover EJB3, datasources, etc). But that parent instance was created within the RESOURCE INSTANCE CLASSLOADER. We are in the context of the PLUGIN classloader. So, the class (even though its the same name class) is actually different. So, the flaw here is any time a parent resource component has its own classloader, the child discovery component can't ever use it (because the classloaders will be different).
--- Possible solution ---
can't the discovery component change its context classloader if it knows it needs to get the parent resource for its connection? In other words, this discovery component knows it needs to talk to its parent resource, and it knows it doesn't have the classes that the parent has, but it DOES know (or we can give it that knowledge if need be) what that RESOURCE INSTANCE classloader is so it can switch to it:
Thread.currentThread.setContextClassLoader(discoveryContext.getParentResourceComponent().getClass().getClassLoader()).
That getParentResourceComponent() is the thing throwing the exception I think, so we may have to add a new getter to the discovery context to help - getParentResourceComponentClassLoader():
Thread.currentThread.setContextClassLoader(discoveryContext.getParentResourceComponentClassLoader());
... now it might be able to successfully call ManagementView ....
--- End Possible Solution ---
Anyway, that's what's going on now with this classloader stuff. This is typical in how this stuff has been going... I advance the implementation some, hit a design flaw, fix the design flaw, further advance the implementation, hit the next design flaw... rinse repeat. I'm just in interation #6 now, I believe :)
15 years, 5 months
classloader stuff in branches
by mazz＠redhat.com
OK, I'm close, but not done. I've put stuff in branches - if you actually svn co these two branches, build them the normal way and run them, you will see the problem I describe below. I'm hoping someone can give me some insight into this if possible
First, the branches are:
RHQ: http://svn.rhq-project.org/repos/rhq/branches/AGENT_PLUGIN_CLASSLOADING/
Jopr: https://svn.jboss.org/repos/jopr/branches/INSTANCE_CLASSLOADERS/
If you run a JbossAS 5 instance, deploy the Jopr branch's plugins and then run the RHQ branch's server and agent, you will notice that this line from the jboss-as-5 plugin:
RemoteProfileServiceConnectionProvider [line: 98]:
profileService = (ProfileService) lookup(initialContext, PROFILE_SERVICE_JNDI_NAME);
produces a ClassCastException. It is trying to cast an AOP dynamic proxy (that is a proxy to the interface of ProfileService) and its telling me it can't cast to ProfileService. But it should be able to.
I have a very bad feeling about this. Like the ProfileService interface class was loaded in one class loader, but the ProfileService interface used by the proxy was loaded by another classloader.
While at a breakpoint at line 98, I confirmed that the same classloader instance is returned by a) the current thread's context classloader, b) this.getClass().getClassLoader() (that is, this RemoteProfileServiceConnectionProvider instance) and c) ProfileService.class.getClassLoader(). So that's all goodness. Haven't figured out yet the classloader of the AOP proxy's interface (can't get to it from the debugger).
I haven't had a chance to delve any deeper other than to actually find this problem (I've solved many other problems just to get to this point). I'll try to do so this weekend, but I was hoping maybe someone saw this kind of thing before or knows the problem?
John
15 years, 5 months