On 02/25/2014 09:26 PM, James Livingston wrote: > Hi, > > I do support for JBoss at RH, especially lower level components like the > JVM and core server, and I've come to realise that we don't have > particularly good mechanisms for gathering diagnostic data. Obviously > there is logging, but for a number of problems it isn't enough. There is > also the JDR subsystem, but that has the serious limitation that the > server instance(s) have to be working correctly to use it. > > > One piece of data that I think can be useful for a common set of > problems are thread dumps. For standalone instances Oracle/OpenJDK's > 'jstack' tool is useful, but for domain mode you need to manually go to > each machine and find the correct process ID to use it. Something that > could potentially be useful is to be able to tell Host Controllers to > take a thread dump from the servers it manages (or all of them), using > the same Attach/SA APIs as jstack. > > A feature to allow HC-based extensions as mentioned in the thread about > the RHQ agent could be useful for implementing this too. Right now, I > believe it would need to be added to the HC code directly. > > > Is this something that might be possible or a good idea to do? I have > some ideas about other things that may be useful, like per-thread CPU > usage which RH support currently uses a pile of shell scripts for, but > starting with one thing may be simpler. I believe that the management model that is already defined for host controllers' views of their servers could easily accommodate this kind of thing. I don't think it'd have to be an extension - in fact, it probably should not be in this case.

On Tue, 2014-02-25 at 22:11 -0600, Brian Stansberry wrote: > On 2/25/14, 9:30 PM, David M. Lloyd wrote: >> On 02/25/2014 09:26 PM, James Livingston wrote: >>> Is this something that might be possible or a good idea to do? I have >>> some ideas about other things that may be useful, like per-thread CPU >>> usage which RH support currently uses a pile of shell scripts for, but >>> starting with one thing may be simpler. >> >> I believe that the management model that is already defined for host >> controllers' views of their servers could easily accommodate this kind >> of thing. I don't think it'd have to be an extension - in fact, it >> probably should not be in this case. >> > > Agreed. Thanks David and Brian. I'll take a look at adding it there, using my in-progress code to abstract over the JDK/OS specific details of grabbing the data. I've created https://issues.jboss.org/browse/WFLY-3025 for it. The most common mechanism for getting a thread dump is on the Hotspot-specific implementation of the Attach API, which (unfortunately) returns a blob of text. Parsing it isn't too hard, but it isn't standardised and newer JDK releases may emit it in a slightly different form. Would it be best to just return the dump as a giant string, or attempt to returning it as a well structured model, with a fallback if parsing fails. The latter would be better for tools, such as if the console ever wanted to do anything with the information, but we have to deal failures to parse the data.

On Wed, 2014-02-26 at 06:41 -0600, Brian Stansberry wrote: > You either need to provide 2 ops, or the details with no fallback, or > just the unstructured text. It can't be a single op that provides well > structured data except if something goes wrong in which case it provides > unstructured text. The metadata for an op describes the structure of the > return data, and that description can't be ambiguous. > > I think starting with a just-text op is best and then let your users > help drive your roadmap. Thanks, I'll do that and see what everyone wants to use it for. To me, the obvious place for these kind of server-specific ops in domain mode would be under /host=*/server=*/ probably under sub-model

since there could be a non-trivial number of useful operations. Operations there get routed to the server to execute, but I need code run on the HC instead.

I've seen the HOST_CONTROLLER_ONLY flag which sounds like what I want, but that doesn't affect /host/server ops. Am I mis-understanding the flag and that's not what it's for, or should it work?

PrepareStepHandler uses isServerOperation() to direct operations under a server to the instance without regard for HOST_CONTROLLER_ONLY, and OperationRouting.determineRouting() only checks the flag if it's not in that location.

in-line... On 3/5/14, 3:34 PM, Brian Stansberry wrote: > On 3/4/14, 6:49 PM, James Livingston wrote: >> To me, the obvious place for these kind of server-specific ops in domain >> mode would be under /host=*/server=*/ probably under sub-model > > Doing this in a sub-resource would be very difficult as it significantly > complicates the issues I'll discuss below. > >> since >> there could be a non-trivial number of useful operations. Operations >> there get routed to the server to execute, but I need code run on the HC >> instead. >> > > Yes, this is a general problem; the /host=*/server=* resource actually > resides on the server process; on the host process at that address all > there is is some proxy code that proxies the request to the server. > > If you look at DomainModelControllerService.registerRunningServer() you > can see where it adds some extra ops to the resource registration for > the server address: > > // Register local operation overrides > final ManagementResourceRegistration serverRegistration = > hostRegistration.getSubModel(PathAddress.EMPTY_ADDRESS.append(pe)); > > ServerConfigResourceDefinition.registerServerLifecycleOperations(serverRegistration, > > serverInventory); > > That's ok for that single case, but we need to find a slicker mechanism > for doing this kind of thing. I don't want DomainModelControllerService > to end up full of this kind of logic. > > If you want to add in another call like that > ServerConfigResourceDefinition.registerServerLifecycleOperations call > for now to keep progressing though, that's fine. We'll just need to come > up with something better before we're done. > Looking at ConcreteResourceRegistration.registerProxyController, what happens is it ignores the existing child registration at server=* and goes ahead and registers the proxy controller because it's address is server=foo. So thereafter "foo" essentially masks "*" when the request is for "foo". If this worked more like the override model stuff instead of a complete mask, then operations registered for "*" would be visible. StoppedServerResource is what currently registers these. WDYT?

Yes, this is a general problem; the /host=*/server=* resource actually resides on the server process; on the host process at that address all there is is some proxy code that proxies the request to the server.

That's ok for that single case, but we need to find a slicker mechanism for doing this kind of thing. I don't want DomainModelControllerService to end up full of this kind of logic. If you want to add in another call like that ServerConfigResourceDefinition.registerServerLifecycleOperations call for now to keep progressing though, that's fine. We'll just need to come up with something better before we're done.

From: "Brian Stansberry" <brian.stansberry(a)redhat.com&gt; To: wildfly-dev(a)lists.jboss.org Sent: Tuesday, February 25, 2014 9:11:15 PM Subject: Re: [wildfly-dev] Gathering diagnostic data On 2/25/14, 9:30 PM, David M. Lloyd wrote: > On 02/25/2014 09:26 PM, James Livingston wrote: >> Hi, >> >> I do support for JBoss at RH, especially lower level components >> like the >> JVM and core server, and I've come to realise that we don't have >> particularly good mechanisms for gathering diagnostic data. >> Obviously >> there is logging, but for a number of problems it isn't enough. >> There is >> also the JDR subsystem, but that has the serious limitation that >> the >> server instance(s) have to be working correctly to use it. >> >> >> One piece of data that I think can be useful for a common set of >> problems are thread dumps. For standalone instances >> Oracle/OpenJDK's >> 'jstack' tool is useful, but for domain mode you need to manually >> go to >> each machine and find the correct process ID to use it. Something >> that >> could potentially be useful is to be able to tell Host Controllers >> to >> take a thread dump from the servers it manages (or all of them), >> using >> the same Attach/SA APIs as jstack. >> >> A feature to allow HC-based extensions as mentioned in the thread >> about >> the RHQ agent could be useful for implementing this too. Right >> now, I >> believe it would need to be added to the HC code directly. >> >> >> Is this something that might be possible or a good idea to do? I >> have >> some ideas about other things that may be useful, like per-thread >> CPU >> usage which RH support currently uses a pile of shell scripts for, >> but >> starting with one thing may be simpler. > > I believe that the management model that is already defined for > host > controllers' views of their servers could easily accommodate this > kind > of thing. I don't think it'd have to be an extension - in fact, it > probably should not be in this case. > > Agreed. -- Brian Stansberry Senior Principal Software Engineer JBoss by Red Hat _______________________________________________ wildfly-dev mailing list wildfly-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/wildfly-dev