[jboss-jira] [JBoss JIRA] (WFLY-11031) Json encoding of Access Log Events

[Brian Stansberry (JIRA)]

    [ https://issues.jboss.org/browse/WFLY-11031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13634417#comment-13634417 ] 

Brian Stansberry commented on WFLY-11031:
-----------------------------------------

Thanks [~fschmager] for the report.  JSON logging is something we've been thinking about in the background a bit all summer and this is good input. An access log is different from a general server log in that the message has a usable structure. It seems obvious now that you mentioned it! 

IIRC there may be a consumer side issue of some sort with nested structures; i.e. reading a "message" field that is itself complex JSON structure as opposed to a simple string.  But I could be misremembering or that whatever is was might be fixed. We'll find out when this gets taken up.

> Json encoding of Access Log Events
> ----------------------------------
>
>                 Key: WFLY-11031
>                 URL: https://issues.jboss.org/browse/WFLY-11031
>             Project: WildFly
>          Issue Type: Feature Request
>          Components: Web (Undertow)
>            Reporter: Frank Schmager
>            Assignee: Stuart Douglas
>            Priority: Minor
>
> h1. Request
> It should be possible to emit access log events json encoded to allow log aggregation systems to ingest pre-parsed events for filtering and analytics.
> h1. Background
> Application logs can be emitted [json encoded|https://wildscribe.github.io/WildFly/13.0/subsystem/logging/json-formatter/index.html] allowing log aggregation systems like logstash/ELK to ingest and parse the events. This is especially important when working with containers, but anything requiring log aggregation would benefit. Not all log aggregation systems allow after-the-fact parsing of log events (e.g. Kibana) and developers don't have access to the log aggregator to install a parser like logstash/filebeat.
> [Access log events|https://wildscribe.github.io/WildFly/13.0/subsystem/undertow/server/host/setting/access-log/index.html], however, can't be json encoded. The whole log event can be, but what's important about access log events can't.
> h2. Example result
> There are many ways to encode the event into json. Ideally it would be configurable. One way would be to use {{message}}, but instead of it just containing a {{String}} it could contain json. However, abusing the {{message}} field might not work because it is of type {{String|text}}. So even if it contained valid json, it wouldn't be processed as such. Separate fields might need to be used. Using {{message}} is probably preferred because if not, what else would that field contain?
> {code:json}
> ...
> "@timestamp": "2018-09-14T20:59:37.530Z",
> ...
> "message": {
>   "x-forwarded-for": "-",
>   "remoteHost": "172.20.2.1",
>   ...
> }
> ...
> # alternatively
> "x-forwarded-for": "-",
> "remoteHost": "172.20.2.1",
> "remoteUser": "200",
> "dateTime": "[14/Sep/2018:20:59:37 +0000]",
> "requestLine": "GET /auth/resources/4.4.0.final/login/keycloak/img/keycloak-bg.png HTTP/2.0",
> "status": "200",
> "bytesSent": "81862",
> "timeTaken": "10",
> "userAgent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36"
> ...
> {code} 
> h2. Attempted Workaround
> Making the pattern look like json failed due to double quote escaping:
> {code:xml}
> <access-log pattern='{\"x-forwarded-for\":\"%{i,X-Forwarded-For}\",\"remoteHost\":\"%h\",\"remoteUser\":\"%s\",\"dateTime\":\"%t\",\"requestLine\":\"%r\",\"status\":\"%s\",\"bytesSent\":\"%b\",\"timeTaken\":\"%D\",\"userAgent\":\"%{i,User-Agent}\"}' use-server-log="true" />
> {code}
> h3. Result
> {code}
> ...
> "@timestamp": "2018-09-14T20:59:37.530Z",
> ...
> "message": "{\\\"x-forwarded-for\\\":\\\"-\\\",\\\"remoteHost\\\":\\\"172.20.2.1\\\",\\\"remoteUser\\\":\\\"200\\\",\\\"dateTime\\\":\\\"[14/Sep/2018:20:59:37 +0000]\\\",\\\"requestLine\\\":\\\"GET /auth/resources/4.4.0.final/login/keycloak/img/keycloak-bg.png HTTP/2.0\\\",\\\"status\\\":\\\"200\\\",\\\"bytesSent\\\":\\\"81862\\\",\\\"timeTaken\\\":\\\"10\\\",\\\"userAgent\\\":\\\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36\\\"}",
> {code}
> h2. Json formatting log configuration
> {code:xml}
> <subsystem xmlns="urn:jboss:domain:logging:5.0">
>   <console-handler name="CONSOLE">
>     <level name="DEBUG"/>
>     <formatter>
>       <named-formatter name="JSON_FORMATTER"/>
>     </formatter>
>   </console-handler>
> ...
>   <formatter name="JSON_FORMATTER">
>     <json-formatter>
>       <exception-output-type value="formatted"/>
>       <key-overrides timestamp="@timestamp" loggerName="logger_name"/>
>       <meta-data>
>         <property name="@version" value="1"/>
>       </meta-data>
>     </json-formatter>
>   </formatter>
> ...
> {code}



--
This message was sent by Atlassian JIRA
(v7.5.0#75005)