[jboss-jira] [JBoss JIRA] (WFLY-11031) Json encoding of Access Log Events

Json encoding of Access Log Events ---------------------------------- Key: WFLY-11031 URL: https://issues.jboss.org/browse/WFLY-11031 Project: WildFly Issue Type: Feature Request Components: Web (Undertow) Reporter: Frank Schmager Assignee: Stuart Douglas Priority: Minor h1. Request It should be possible to emit access log events json encoded to allow log aggregation systems to ingest pre-parsed events for filtering and analytics. h1. Background Application logs can be emitted [json encoded|https://wildscribe.github.io/WildFly/13.0/subsystem/logging/json-formatter/index.html] allowing log aggregation systems like logstash/ELK to ingest and parse the events. This is especially important when working with containers, but anything requiring log aggregation would benefit. Not all log aggregation systems allow after-the-fact parsing of log events (e.g. Kibana) and developers don't have access to the log aggregator to install a parser like logstash/filebeat. [Access log events|https://wildscribe.github.io/WildFly/13.0/subsystem/undertow/server/host/setting/access-log/index.html], however, can't be json encoded. The whole log event can be, but what's important about access log events can't. h2. Example result There are many ways to encode the event into json. Ideally it would be configurable. One way would be to use {{message}}, but instead of it just containing a {{String}} it could contain json. However, abusing the {{message}} field might not work because it is of type {{String|text}}. So even if it contained valid json, it wouldn't be processed as such. Separate fields might need to be used. Using {{message}} is probably preferred because if not, what else would that field contain? {code:json} ... "@timestamp": "2018-09-14T20:59:37.530Z", ... "message": { "x-forwarded-for": "-", "remoteHost": "172.20.2.1", ... } ... # alternatively "x-forwarded-for": "-", "remoteHost": "172.20.2.1", "remoteUser": "200", "dateTime": "[14/Sep/2018:20:59:37 +0000]", "requestLine": "GET /auth/resources/4.4.0.final/login/keycloak/img/keycloak-bg.png HTTP/2.0", "status": "200", "bytesSent": "81862", "timeTaken": "10", "userAgent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36" ... {code} h2. Attempted Workaround Making the pattern look like json failed due to double quote escaping: {code:xml} <access-log pattern='{\"x-forwarded-for\":\"%{i,X-Forwarded-For}\",\"remoteHost\":\"%h\",\"remoteUser\":\"%s\",\"dateTime\":\"%t\",\"requestLine\":\"%r\",\"status\":\"%s\",\"bytesSent\":\"%b\",\"timeTaken\":\"%D\",\"userAgent\":\"%{i,User-Agent}\"}' use-server-log="true" /> {code} h3. Result {code} ... "@timestamp": "2018-09-14T20:59:37.530Z", ... "message": "{\\\"x-forwarded-for\\\":\\\"-\\\",\\\"remoteHost\\\":\\\"172.20.2.1\\\",\\\"remoteUser\\\":\\\"200\\\",\\\"dateTime\\\":\\\"[14/Sep/2018:20:59:37 +0000]\\\",\\\"requestLine\\\":\\\"GET /auth/resources/4.4.0.final/login/keycloak/img/keycloak-bg.png HTTP/2.0\\\",\\\"status\\\":\\\"200\\\",\\\"bytesSent\\\":\\\"81862\\\",\\\"timeTaken\\\":\\\"10\\\",\\\"userAgent\\\":\\\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36\\\"}", {code} h2. Json formatting log configuration {code:xml} <subsystem xmlns="urn:jboss:domain:logging:5.0"> <console-handler name="CONSOLE"> <level name="DEBUG"/> <formatter> <named-formatter name="JSON_FORMATTER"/> </formatter> </console-handler> ... <formatter name="JSON_FORMATTER"> <json-formatter> <exception-output-type value="formatted"/> <key-overrides timestamp="@timestamp" loggerName="logger_name"/> <meta-data> <property name="@version" value="1"/> </meta-data> </json-formatter> </formatter> ... {code}