[jboss-jira] [JBoss JIRA] (WFLY-11031) Json encoding of Access Logs
Frank Schmager (JIRA)
issues at jboss.org
Mon Sep 17 09:08:00 EDT 2018
[ https://issues.jboss.org/browse/WFLY-11031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Frank Schmager updated WFLY-11031:
----------------------------------
Description:
h1. Request
It should be possible to emit access log events json encoded to allow log aggregation systems to ingest pre-parsed events for filtering and analytics.
h1. Background
Application logs can be emitted [json encoded|https://wildscribe.github.io/WildFly/13.0/subsystem/logging/json-formatter/index.html] allowing log aggregation systems like logstash/ELK to ingest and parse the events. This is especially important when working with containers, but anything requiring log aggregation would benefit. Not all log aggregation systems allow after-the-fact parsing of log events (e.g. Kibana) and developers don't have access to the log aggregator to install a parser like logstash/filebeat.
[Access log events|https://wildscribe.github.io/WildFly/13.0/subsystem/undertow/server/host/setting/access-log/index.html], however, can't be json encoded. The whole log event can be, but what's important about access log events can't.
h2. Example result
There are many ways to encode the event into json. Ideally it would be configurable. One way would be to use {{message}}, but instead of it just containing a {{String}} it could contain json. However, abusing the {{message}} field might not work because it is of type {{String|text}}. So even if it contained valid json, it wouldn't be processed as such. Separate fields might need to be used. Using {{message}} is probably preferred because if not, what else would that field contain?
{code:json}
...
"@timestamp": "2018-09-14T20:59:37.530Z",
...
"message": {
"x-forwarded-for": "-",
"remoteHost": "172.20.2.1",
...
}
...
# alternatively
"x-forwarded-for": "-",
"remoteHost": "172.20.2.1",
"remoteUser": "200",
"dateTime": "[14/Sep/2018:20:59:37 +0000]",
"requestLine": "GET /auth/resources/4.4.0.final/login/keycloak/img/keycloak-bg.png HTTP/2.0",
"status": "200",
"bytesSent": "81862",
"timeTaken": "10",
"userAgent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36"
...
{code}
h2. Attempted Workaround
Making the pattern look like json failed due to double quote escaping:
{code:xml}
<access-log pattern='{\"x-forwarded-for\":\"%{i,X-Forwarded-For}\",\"remoteHost\":\"%h\",\"remoteUser\":\"%s\",\"dateTime\":\"%t\",\"requestLine\":\"%r\",\"status\":\"%s\",\"bytesSent\":\"%b\",\"timeTaken\":\"%D\",\"userAgent\":\"%{i,User-Agent}\"}' use-server-log="true" />
{code}
h3. Result
{code}
...
"@timestamp": "2018-09-14T20:59:37.530Z",
...
"message": "{\\\"x-forwarded-for\\\":\\\"-\\\",\\\"remoteHost\\\":\\\"172.20.2.1\\\",\\\"remoteUser\\\":\\\"200\\\",\\\"dateTime\\\":\\\"[14/Sep/2018:20:59:37 +0000]\\\",\\\"requestLine\\\":\\\"GET /auth/resources/4.4.0.final/login/keycloak/img/keycloak-bg.png HTTP/2.0\\\",\\\"status\\\":\\\"200\\\",\\\"bytesSent\\\":\\\"81862\\\",\\\"timeTaken\\\":\\\"10\\\",\\\"userAgent\\\":\\\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36\\\"}",
{code}
h2. Json formatting log configuration
{code:xml}
<subsystem xmlns="urn:jboss:domain:logging:5.0">
<console-handler name="CONSOLE">
<level name="DEBUG"/>
<formatter>
<named-formatter name="JSON_FORMATTER"/>
</formatter>
</console-handler>
...
<formatter name="JSON_FORMATTER">
<json-formatter>
<exception-output-type value="formatted"/>
<key-overrides timestamp="@timestamp" loggerName="logger_name"/>
<meta-data>
<property name="@version" value="1"/>
</meta-data>
</json-formatter>
</formatter>
...
{code}
was:
h1. Request
It should be possible to emit access log events json encoded to allow log aggregation systems to ingest pre-parsed events for filtering and analytics.
h1. Background
Application logs can be emitted [json encoded|https://wildscribe.github.io/WildFly/13.0/subsystem/logging/json-formatter/index.html] allowing log aggregation systems like logstash/ELK to ingest and parse the events. This is especially important when working with containers, but anything requiring log aggregation would benefit. Not all log aggregation systems allow after-the-fact parsing of log events (e.g. Kibana) and developers don't have access to the log aggregator to install a parser like logstash/filebeat.
[Access log events|https://wildscribe.github.io/WildFly/13.0/subsystem/undertow/server/host/setting/access-log/index.html], however, can't be json encoded. The whole log event can be, but what's important about access log events can't.
h2. Example result
There are many ways to encode the event into json. Ideally it would be configurable. One way would be to use {{message}}, but instead of it just containing a {{String}} it could contain json.
{code:json}
...
"@timestamp": "2018-09-14T20:59:37.530Z",
...
"message": {
"x-forwarded-for": "-",
"remoteHost": "172.20.2.1",
"remoteUser": "200",
"dateTime": "[14/Sep/2018:20:59:37 +0000]",
"requestLine": "GET /auth/resources/4.4.0.final/login/keycloak/img/keycloak-bg.png HTTP/2.0",
"status": "200",
"bytesSent": "81862",
"timeTaken": "10",
"userAgent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36"
}
{code}
h2. Attempted Workaround
Making the pattern look like json failed due to double quote escaping:
{code:xml}
<access-log pattern='{\"x-forwarded-for\":\"%{i,X-Forwarded-For}\",\"remoteHost\":\"%h\",\"remoteUser\":\"%s\",\"dateTime\":\"%t\",\"requestLine\":\"%r\",\"status\":\"%s\",\"bytesSent\":\"%b\",\"timeTaken\":\"%D\",\"userAgent\":\"%{i,User-Agent}\"}' use-server-log="true" />
{code}
h3. Result
{code}
...
"@timestamp": "2018-09-14T20:59:37.530Z",
...
"message": "{\\\"x-forwarded-for\\\":\\\"-\\\",\\\"remoteHost\\\":\\\"172.20.2.1\\\",\\\"remoteUser\\\":\\\"200\\\",\\\"dateTime\\\":\\\"[14/Sep/2018:20:59:37 +0000]\\\",\\\"requestLine\\\":\\\"GET /auth/resources/4.4.0.final/login/keycloak/img/keycloak-bg.png HTTP/2.0\\\",\\\"status\\\":\\\"200\\\",\\\"bytesSent\\\":\\\"81862\\\",\\\"timeTaken\\\":\\\"10\\\",\\\"userAgent\\\":\\\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36\\\"}",
{code}
h2. Json formatting log configuration
{code:xml}
<subsystem xmlns="urn:jboss:domain:logging:5.0">
<console-handler name="CONSOLE">
<level name="DEBUG"/>
<formatter>
<named-formatter name="JSON_FORMATTER"/>
</formatter>
</console-handler>
...
<formatter name="JSON_FORMATTER">
<json-formatter>
<exception-output-type value="formatted"/>
<key-overrides timestamp="@timestamp" loggerName="logger_name"/>
<meta-data>
<property name="@version" value="1"/>
</meta-data>
</json-formatter>
</formatter>
...
{code}
> Json encoding of Access Logs
> ----------------------------
>
> Key: WFLY-11031
> URL: https://issues.jboss.org/browse/WFLY-11031
> Project: WildFly
> Issue Type: Feature Request
> Components: Web (Undertow)
> Reporter: Frank Schmager
> Assignee: Stuart Douglas
> Priority: Minor
>
> h1. Request
> It should be possible to emit access log events json encoded to allow log aggregation systems to ingest pre-parsed events for filtering and analytics.
> h1. Background
> Application logs can be emitted [json encoded|https://wildscribe.github.io/WildFly/13.0/subsystem/logging/json-formatter/index.html] allowing log aggregation systems like logstash/ELK to ingest and parse the events. This is especially important when working with containers, but anything requiring log aggregation would benefit. Not all log aggregation systems allow after-the-fact parsing of log events (e.g. Kibana) and developers don't have access to the log aggregator to install a parser like logstash/filebeat.
> [Access log events|https://wildscribe.github.io/WildFly/13.0/subsystem/undertow/server/host/setting/access-log/index.html], however, can't be json encoded. The whole log event can be, but what's important about access log events can't.
> h2. Example result
> There are many ways to encode the event into json. Ideally it would be configurable. One way would be to use {{message}}, but instead of it just containing a {{String}} it could contain json. However, abusing the {{message}} field might not work because it is of type {{String|text}}. So even if it contained valid json, it wouldn't be processed as such. Separate fields might need to be used. Using {{message}} is probably preferred because if not, what else would that field contain?
> {code:json}
> ...
> "@timestamp": "2018-09-14T20:59:37.530Z",
> ...
> "message": {
> "x-forwarded-for": "-",
> "remoteHost": "172.20.2.1",
> ...
> }
> ...
> # alternatively
> "x-forwarded-for": "-",
> "remoteHost": "172.20.2.1",
> "remoteUser": "200",
> "dateTime": "[14/Sep/2018:20:59:37 +0000]",
> "requestLine": "GET /auth/resources/4.4.0.final/login/keycloak/img/keycloak-bg.png HTTP/2.0",
> "status": "200",
> "bytesSent": "81862",
> "timeTaken": "10",
> "userAgent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36"
> ...
> {code}
> h2. Attempted Workaround
> Making the pattern look like json failed due to double quote escaping:
> {code:xml}
> <access-log pattern='{\"x-forwarded-for\":\"%{i,X-Forwarded-For}\",\"remoteHost\":\"%h\",\"remoteUser\":\"%s\",\"dateTime\":\"%t\",\"requestLine\":\"%r\",\"status\":\"%s\",\"bytesSent\":\"%b\",\"timeTaken\":\"%D\",\"userAgent\":\"%{i,User-Agent}\"}' use-server-log="true" />
> {code}
> h3. Result
> {code}
> ...
> "@timestamp": "2018-09-14T20:59:37.530Z",
> ...
> "message": "{\\\"x-forwarded-for\\\":\\\"-\\\",\\\"remoteHost\\\":\\\"172.20.2.1\\\",\\\"remoteUser\\\":\\\"200\\\",\\\"dateTime\\\":\\\"[14/Sep/2018:20:59:37 +0000]\\\",\\\"requestLine\\\":\\\"GET /auth/resources/4.4.0.final/login/keycloak/img/keycloak-bg.png HTTP/2.0\\\",\\\"status\\\":\\\"200\\\",\\\"bytesSent\\\":\\\"81862\\\",\\\"timeTaken\\\":\\\"10\\\",\\\"userAgent\\\":\\\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36\\\"}",
> {code}
> h2. Json formatting log configuration
> {code:xml}
> <subsystem xmlns="urn:jboss:domain:logging:5.0">
> <console-handler name="CONSOLE">
> <level name="DEBUG"/>
> <formatter>
> <named-formatter name="JSON_FORMATTER"/>
> </formatter>
> </console-handler>
> ...
> <formatter name="JSON_FORMATTER">
> <json-formatter>
> <exception-output-type value="formatted"/>
> <key-overrides timestamp="@timestamp" loggerName="logger_name"/>
> <meta-data>
> <property name="@version" value="1"/>
> </meta-data>
> </json-formatter>
> </formatter>
> ...
> {code}
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
More information about the jboss-jira
mailing list