[wildfly-dev] Steamable content in management API responses

tl;dr I'm looking into supported streamed content in management API responses and wanted to get feedback. The admin console has a need for a streamed response, as we have a requirement to let a user download a log file to store on local disk. The browser can itself directly download a file using HTTP, but will not let an app like the console make multiple DMR requests and itself append data from the responses to a file on disk. There are other likely uses cases for a similar thing, e.g. reading the contents of the content (aka deployment) repository. We already support attaching streams to requests; the proposal is to do much the same with responses. For HTTP clients, if a request is for something that attaches a stream to the response, the request URL can include a query param instructing the server to pipe the stream for the response instead of sending the standard JSON reponse value. Long version: Requirements: 1) Ability to send arbitrarily long amounts of data in management responses. Currently everything in a management response has to be encodable in DMR. DMR will allow you to send a byte[] as a response value, but for memory reasons this isn't a practical approach for large responses. 2) Ability to send the data directly to an HTTP request, not wrapped in a JSON wrapper. Even if we would include a huge value as a JSON encoded byte[], a browser won't let an app like the console read those bytes out of the JSON response and write them to disk. A direct response to an HTTP GET/POST is needed. 3) Works for other remoting-based clients as well. This is a basic requirement that HTTP and remoting-based clients can each do everything. So, no HTTP only side features. 4) Requests go through the central management layer, so we have proper security, audit logging etc. So, no special endpoints that bypass the core management layer. Proposal: We already support attaching a stream to the request for native clients. (This is how the CLI does deployments.) The client API lets the client associate a stream with the request[1]. The stream has an index. The operation that needs the stream takes a DMR param that tells it the index of the stream. On the server side the handler uses that param value to find the server-side representation of the stream. The remoting layer of the remote management protocol in the background handles piping the contents of the client side stream to the server. The proposal is to simply mirror this in reverse. A server side operation handler associates a stream with the response (via a call on the OperationContext). The attached stream has an index. The normal DMR response to the operation is the index of the stream. The client uses that response value to find the attached stream. The remoting layer of the remote management protocol in the background handles piping the contents of the server side stream to the client. This handles remoting based clients, including those using http upgrade. For HTTP clients, the client can include a "useStreamAsResponse" query param in the request URL. If this is present, the server side endpoint will take the stream from the internal response and pipe it to the client, instead of sending the normal JSON. For example, a fictitious URL for a "stream-log-file" op against a resource that represents the server.log file: http://localhost:9090/management/subsystem/logging/log-file/server.log?op... In the corner case where a request results in more than one attached response, the useStreamAsResponse could include an index. useStreamAsResponse=1 Status: It was reasonably straightforward to get this working, as a fairly polished prototype: https://github.com/bstansberry/wildfly-core/commits/resp-stream The 2nd commit there just hacks in an attribute to the logging subsystem root resource to expose the server log as a stream. There's no intent to actually do it that specific way of course; it was just an easy way to demonstrate. With that built an HTTP GET will download the log file to your desktop. http://localhost:9990/management/subsystem/logging?operation=attribute&am... The native interface works as well. The 2nd commit includes a test case that confirms this. So the CLI easily enough could add a high level command for log reading too. TODOs: 1) Domain mode: proxy the streams around the domain (server->HC->DC->client). I think this should be quite easy. 2) A cleaner variant of the remoting-based management protocol for handling the streams, one that doesn't require hacks to determine the length of the stream in advance. Should be relatively straightforward. 3) Logic to clean up server side streams if the client doesn't properly consume them. 4) Make sure there a no issues with the thread pools used to handle management requests. 5) Make sure POST works well. My assumption is this is lower priority, as the real use cases would likely use a GET. [1] https://github.com/wildfly/wildfly-core/blob/master/controller-client/src... -- Brian Stansberry Senior Principal Software Engineer JBoss by Red Hat