Hi, I do support for JBoss at RH, especially lower level components like the JVM and core server, and I've come to realise that we don't have particularly good mechanisms for gathering diagnostic data. Obviously there is logging, but for a number of problems it isn't enough. There is also the JDR subsystem, but that has the serious limitation that the server instance(s) have to be working correctly to use it. One piece of data that I think can be useful for a common set of problems are thread dumps. For standalone instances Oracle/OpenJDK's 'jstack' tool is useful, but for domain mode you need to manually go to each machine and find the correct process ID to use it. Something that could potentially be useful is to be able to tell Host Controllers to take a thread dump from the servers it manages (or all of them), using the same Attach/SA APIs as jstack. A feature to allow HC-based extensions as mentioned in the thread about the RHQ agent could be useful for implementing this too. Right now, I believe it would need to be added to the HC code directly. Is this something that might be possible or a good idea to do? I have some ideas about other things that may be useful, like per-thread CPU usage which RH support currently uses a pile of shell scripts for, but starting with one thing may be simpler.

On 2/25/14, 9:30 PM, David M. Lloyd wrote: > On 02/25/2014 09:26 PM, James Livingston wrote: >> Is this something that might be possible or a good idea to do? I have >> some ideas about other things that may be useful, like per-thread CPU >> usage which RH support currently uses a pile of shell scripts for, but >> starting with one thing may be simpler. > > I believe that the management model that is already defined for host > controllers' views of their servers could easily accommodate this kind > of thing. I don't think it'd have to be an extension - in fact, it > probably should not be in this case. > Agreed.

From: "Brian Stansberry" <brian.stansberry(a)redhat.com&gt; To: wildfly-dev(a)lists.jboss.org Sent: Tuesday, February 25, 2014 9:11:15 PM Subject: Re: [wildfly-dev] Gathering diagnostic data On 2/25/14, 9:30 PM, David M. Lloyd wrote: > On 02/25/2014 09:26 PM, James Livingston wrote: >> Hi, >> >> I do support for JBoss at RH, especially lower level components >> like the >> JVM and core server, and I've come to realise that we don't have >> particularly good mechanisms for gathering diagnostic data. >> Obviously >> there is logging, but for a number of problems it isn't enough. >> There is >> also the JDR subsystem, but that has the serious limitation that >> the >> server instance(s) have to be working correctly to use it. >> >> >> One piece of data that I think can be useful for a common set of >> problems are thread dumps. For standalone instances >> Oracle/OpenJDK's >> 'jstack' tool is useful, but for domain mode you need to manually >> go to >> each machine and find the correct process ID to use it. Something >> that >> could potentially be useful is to be able to tell Host Controllers >> to >> take a thread dump from the servers it manages (or all of them), >> using >> the same Attach/SA APIs as jstack. >> >> A feature to allow HC-based extensions as mentioned in the thread >> about >> the RHQ agent could be useful for implementing this too. Right >> now, I >> believe it would need to be added to the HC code directly. >> >> >> Is this something that might be possible or a good idea to do? I >> have >> some ideas about other things that may be useful, like per-thread >> CPU >> usage which RH support currently uses a pile of shell scripts for, >> but >> starting with one thing may be simpler. > > I believe that the management model that is already defined for > host > controllers' views of their servers could easily accommodate this > kind > of thing. I don't think it'd have to be an extension - in fact, it > probably should not be in this case. > > Agreed. -- Brian Stansberry Senior Principal Software Engineer JBoss by Red Hat _______________________________________________ wildfly-dev mailing list wildfly-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/wildfly-dev

On Wed, 2014-02-26 at 06:41 -0600, Brian Stansberry wrote: > You either need to provide 2 ops, or the details with no fallback, or > just the unstructured text. It can't be a single op that provides well > structured data except if something goes wrong in which case it provides > unstructured text. The metadata for an op describes the structure of the > return data, and that description can't be ambiguous. > > I think starting with a just-text op is best and then let your users > help drive your roadmap. Thanks, I'll do that and see what everyone wants to use it for. To me, the obvious place for these kind of server-specific ops in domain mode would be under /host=*/server=*/ probably under sub-model

since there could be a non-trivial number of useful operations. Operations there get routed to the server to execute, but I need code run on the HC instead.

I've seen the HOST_CONTROLLER_ONLY flag which sounds like what I want, but that doesn't affect /host/server ops. Am I mis-understanding the flag and that's not what it's for, or should it work?

PrepareStepHandler uses isServerOperation() to direct operations under a server to the instance without regard for HOST_CONTROLLER_ONLY, and OperationRouting.determineRouting() only checks the flag if it's not in that location.

Yes, this is a general problem; the /host=*/server=* resource actually resides on the server process; on the host process at that address all there is is some proxy code that proxies the request to the server.

That's ok for that single case, but we need to find a slicker mechanism for doing this kind of thing. I don't want DomainModelControllerService to end up full of this kind of logic. If you want to add in another call like that ServerConfigResourceDefinition.registerServerLifecycleOperations call for now to keep progressing though, that's fine. We'll just need to come up with something better before we're done.

On Wed, 2014-03-05 at 15:34 -0600, Brian Stansberry wrote: > Yes, this is a general problem; the /host=*/server=* resource actually > resides on the server process; on the host process at that address all > there is is some proxy code that proxies the request to the server. That makes sense, and matches what I was seeing when I was stepping through in a debugger yesterday. I originally thought that it proxied the operations not the whole sub-model. > That's ok for that single case, but we need to find a slicker mechanism > for doing this kind of thing. I don't want DomainModelControllerService > to end up full of this kind of logic. > > If you want to add in another call like that > ServerConfigResourceDefinition.registerServerLifecycleOperations call > for now to keep progressing though, that's fine. We'll just need to come > up with something better before we're done. For now I've just stuck it on as /host=*:thread-dump(server=*), which works fine for developing the more interesting bits. I think these kind of per-server things logically belong under the server, so I'll see what gets thought up in the other branch of the thread.