[wildfly-dev] Reading Logs from Web Console
James R. Perkins
jperkins at redhat.com
Thu Oct 24 10:58:16 EDT 2013
At this point it's hard-coded to only look at the jboss.server.log.dir.
If we need/want something for audit logs I think we might want to do
something separate. It seems like there might be different security
needed to read audit logs. I don't know a lot about them though so
someone please correct me if I'm wrong.
On 10/24/2013 07:01 AM, Harald Pehl wrote:
> Is it possible to read log files not stored under
> ${jboss.server.log.dir}? I'm asking because the audit log is per
> default stored under ${jboss.server.data.dir}
>
> ---
> Harald Pehl
> JBoss by Red Hat
> http://hpehl.info
>
>
>
>
> Am 09.10.2013 um 18:10 schrieb James R. Perkins <jperkins at redhat.com
> <mailto:jperkins at redhat.com>>:
>
>> In my opinion we shouldn't offer any kind of log parsing options. We
>> should simple deliver the raw log file and have the end user use
>> their own parsing. There is a JIRA [1] to allow custom formatters to
>> be used. This means an end user could use an XML formatter and it
>> could be easily parsed by them. Since format patterns are so
>> configurable and we're going to expose the ability to override the
>> formatter we need to just give them the raw file.
>>
>> I suppose saying all that contradicts my reasoning for using the
>> BufferedReader.readLine() and maybe we should be returning by number
>> of bytes/chars instead of by line. There seems to be no real good or
>> solid answer and it would be nice to hear from any users that have an
>> opinion on how they might use it. From the web console I'm going to
>> guess either approach, by line or by bytes, will work fine as it
>> might just construct one large file for download. If using it from a
>> console just to see the last 20 lines of the log file, a line-by-line
>> approach to me makes more sense.
>>
>> [1]: https://issues.jboss.org/browse/WFLY-1188
>>
>> On 10/09/2013 05:09 AM, Ondrej Zizka wrote:
>>> How about storing the offsets of the records.
>>> Then the operation would read this "index" and instead of
>>> format-based parsing, it would simply cut the right part.
>>> Simple, fast in both write and read, and doesn't consume too much
>>> memory/storage.
>>>
>>> Ondra
>>>
>>>
>>> On 8.10.2013 18:57, James R. Perkins wrote:
>>>> I'm definitely not trying to parse any of the log message format. I am
>>>> using a BufferedReader.readLine() which I was a little iffy on, but
>>>> went
>>>> with "lines" as opposed to reading x bytes. Obviously from a
>>>> management
>>>> standpoint reading the number of bytes makes more sense as we're not
>>>> relying on any kind of normal line terminator being used. From a user
>>>> standpoint though I think it makes less sense and the number of lines
>>>> makes more sense.
>>>>
>>>> I suppose we could have two operations so the web console could use
>>>> the
>>>> one that reads bytes and then have one a simple tail like one that
>>>> reads
>>>> by line.
>>>>
>>>> On 10/08/2013 09:26 AM, David M. Lloyd wrote:
>>>>> I've said it before and I'll say it again. Using the format to
>>>>> "intelligently" parse the logs looks like a good idea. It is
>>>>> actually a
>>>>> bad idea. A bad idea that looks like a good idea, or maybe even a
>>>>> "good
>>>>> enough" idea. Maybe it looks like an "easy" idea. But it's bad.
>>>>> Bad,
>>>>> and also not good. It will result in bug reports that nobody will
>>>>> bother fixing or implementing because they will be reasonable yet
>>>>> unfeasible to fix.
>>>>>
>>>>> Here are some implementation options that I *won't* immediately kill:
>>>>>
>>>>> 1) Treat the files in the log directory as opaque things that the
>>>>> server
>>>>> does not pretend to have interior knowledge of - in other words,
>>>>> provide
>>>>> operations that work on the file as a whole (or by things which make
>>>>> sense at a file level, like line number or dumb string grep) which
>>>>> will
>>>>> work 100% of the time
>>>>> 2) Provide in-memory (object) or on-disk (structured binary)
>>>>> storage of
>>>>> log record objects with intelligent search capabilities which will
>>>>> work
>>>>> 100% of the time
>>>>>
>>>>> Here are some implementation options that I *will* immediately kill:
>>>>>
>>>>> 1) Try to use the format to figure out record boundaries
>>>>> 2) Try to use the format to figure out the date of the record
>>>>> 3) Try to use the format to extract exception information
>>>>> 4) Any and all other format-based ideas not mentioned here
>>>>>
>>>>> The common theme is - it has to work 100% of the time. No excuses.
>>>>>
>>>>> On 10/08/2013 10:55 AM, James R. Perkins wrote:
>>>>>> On 10/08/2013 03:40 AM, Kabir Khan wrote:
>>>>>>> Just a thought regarding your PR
>>>>>>> https://github.com/wildfly/wildfly/pull/5156
>>>>>>>
>>>>>>> The thing that stands out to me is that you're able to read the
>>>>>>> first 100 lines, e.g.
>>>>>>> :read-log-file(name=server.log, lines=100, tail=false)
>>>>>>> or the last 100 lines
>>>>>>> :read-log-file(name=server.log, lines=100, tail=true)
>>>>>>>
>>>>>>> Would it also be beneficial to also be allowed to start
>>>>>>> somewhere in the middle, e.g
>>>>>>> :read-log-file(name=server.log, lines=100, start=10235)
>>>>>> There is an option skip that will allows something similar. The
>>>>>> following would read read the last 100 lines _after_ 50 lines are
>>>>>> skipped from the bottom. So in a 200 line this would read lines
>>>>>> 50 to
>>>>>> line 150.
>>>>>> :read-log-file(name=server.log, lines=100, skip=50,
>>>>>> tail=true)
>>>>>>
>>>>>> The following would read from line 50 to line 150
>>>>>> :read-log-file(name=server.log, lines=100, skip=50,
>>>>>> tail=false)
>>>>>>
>>>>>> I am using a BufferedReader so if an odd line terminator is used
>>>>>> it will
>>>>>> likely not work so well. I considered allowing for the line
>>>>>> terminator
>>>>>> to be defined in the operation, but it would require analyzing
>>>>>> bytes for
>>>>>> the terminator and seemed like more work than it was worth. I would
>>>>>> guess the majority of users use the standard /n or /r/n terminators.
>>>>>>> or (19:38:01 today)
>>>>>>> :read-log-file(name=server.log, lines=100,
>>>>>>> start-time=20131008-10:38:01)
>>>>>> The date would be rather difficult. We're not parsing the log
>>>>>> strings
>>>>>> just reading the raw text and sending it back.
>>>>>>> I'm not saying it MUST be like that but it might be worth
>>>>>>> discussing
>>>>>>>
>>>>>>> On 25 Sep 2013, at 16:36, Ondrej Zizka wrote:
>>>>>>>
>>>>>>>> Right, by "current" I meant to read the logging setup and if
>>>>>>>> there's
>>>>>>>> just one file appender, take it as default. Or, if that would be
>>>>>>>> inappropriate mixing of abstraction layers, just "server.log".
>>>>>>>>
>>>>>>>>
>>>>>>>> On 25.9.2013 12:06, Kabir Khan wrote:
>>>>>>>>> Current log could be the server.log in a standard setup. If
>>>>>>>>> someone has done something more advanced in their logging
>>>>>>>>> setup, the name becomes necessary
>>>>>>>>> On 25 Sep 2013, at 08:34, Jaikiran Pai wrote:
>>>>>>>>>
>>>>>>>>>> On Wednesday 25 September 2013 09:25 AM, Ondrej Zizka wrote:
>>>>>>>>>>> 1) Could it have a "read-log-file()" without name=
>>>>>>>>>>> specified, which would read the "current" log file?
>>>>>>>>>> Given the way logger categories and appenders/handlers
>>>>>>>>>> interact, within a logging framework, I don't think there's
>>>>>>>>>> any notion of "current log file". It's a very valid scenario
>>>>>>>>>> where a single logging category can be backed by different
>>>>>>>>>> appenders (some of them file appenders) with different
>>>>>>>>>> attributes and each such appender writing out to a different
>>>>>>>>>> file. So having a name of the log file you want to view,
>>>>>>>>>> becomes necessary.
>>>>>>>>>>
>>>>>>>>>> -Jaikiran
>>>>>>>>>>
>>>>>>>>>>> 2) Regarding security - what, besides logs, do we expect to
>>>>>>>>>>> be in the log dir? Could the admin block it by setting
>>>>>>>>>>> write-only rights?
>>>>>>>>>>>
>>>>>>>>>>> Ondra
>>>>>>>>>>>
>>>>>>>>>>> On 25.9.2013 02:40, James R. Perkins wrote:
>>>>>>>>>>>> I'm replying to this old thread to reopen this conversation
>>>>>>>>>>>> about reading log files. I've complete some work [1] on
>>>>>>>>>>>> reading log files via an operation. This is not exactly
>>>>>>>>>>>> like the JIRA suggests where it would only read the last 10
>>>>>>>>>>>> error messages. All this change allows is the raw contents
>>>>>>>>>>>> of the file to be read. The idea is this could be used to
>>>>>>>>>>>> read the entire contents of the log file as a whole, or in
>>>>>>>>>>>> chunks.
>>>>>>>>>>>>
>>>>>>>>>>>> What I've done is added two new operations list-log-files
>>>>>>>>>>>> and read-log-file.
>>>>>>>>>>>>
>>>>>>>>>>>> The list-log-files simply lists all files in the
>>>>>>>>>>>> jboss.server.log.dir. This may or may not be a good idea
>>>>>>>>>>>> really. I can see some potential security risks here mainly
>>>>>>>>>>>> just seeing files that may contain sensitive data. One way
>>>>>>>>>>>> I've thought of to get around that is read the logging
>>>>>>>>>>>> subsystem model and only show files from known types like
>>>>>>>>>>>> the file-handlers. The main issue with that is there is no
>>>>>>>>>>>> good way to get this to work for custom-handlers.
>>>>>>>>>>>>
>>>>>>>>>>>> The read-log-file simple does what it says and reads the
>>>>>>>>>>>> contents of a log file line by line. Reading line by line
>>>>>>>>>>>> should work for the most part unless the an non-standard
>>>>>>>>>>>> line delimiter is used. There are 5 options for this option;
>>>>>>>>>>>> • name (required): the name of the log file to read
>>>>>>>>>>>> • encoding: the encoding for the log file
>>>>>>>>>>>> • lines: the number of lines to read, defaults to 10
>>>>>>>>>>>> • skip: the number of lines to skip before adding the
>>>>>>>>>>>> results
>>>>>>>>>>>> • tail: true to read from the bottom up, default is true
>>>>>>>>>>>> The result of this is just a list of lines with the \n or
>>>>>>>>>>>> \r\n stripped. Just to clarify too a line means a line in
>>>>>>>>>>>> the file, not a log record e.g. stack traces are generally
>>>>>>>>>>>> composed of multiple lines.
>>>>>>>>>>>>
>>>>>>>>>>>> So this begs the question, will this work for what we want?
>>>>>>>>>>>> What concerns does anyone else have?
>>>>>>>>>>>>
>>>>>>>>>>>> I have not yet submitted a PR yet as I wanted to get some
>>>>>>>>>>>> feedback before we bake it in.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> [1]: https://github.com/jamezp/wildfly/compare/WFLY-280-read
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On 08/14/2013 10:03 AM, James R. Perkins wrote:
>>>>>>>>>>>>> I had posted this to another list, but this is a more
>>>>>>>>>>>>> appropriate place for it. I think there needs to be a
>>>>>>>>>>>>> general discussion around this as it's been mentioned, at
>>>>>>>>>>>>> least to me, a few times here and there and I know Heiko
>>>>>>>>>>>>> raised the issue some time a go now.
>>>>>>>>>>>>>
>>>>>>>>>>>>> The original JIRA, WFLY-280[1], is to display the last 10
>>>>>>>>>>>>> error messages only. To be honest I wouldn't find that
>>>>>>>>>>>>> very useful. To me if I'm looking for logs I want to see
>>>>>>>>>>>>> all logs, but that's not always so easy. Like the
>>>>>>>>>>>>> syslog-handler which doesn't log to a file so there is no
>>>>>>>>>>>>> way to read those messages back.
>>>>>>>>>>>>>
>>>>>>>>>>>>> The current plan for the last 10 error messages is we
>>>>>>>>>>>>> store messages in a queue that can be accessed via an
>>>>>>>>>>>>> operation. This works fine until the error message you're
>>>>>>>>>>>>> interested in is 11 or you want to see warning messages.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Another option I had come up with is reading back the
>>>>>>>>>>>>> contents of the file, for example the server.log. This
>>>>>>>>>>>>> could be problematic too in that there is no way to filter
>>>>>>>>>>>>> information like only see error messages or only see
>>>>>>>>>>>>> warning messages. To solve this I have considered creating
>>>>>>>>>>>>> a JSON formatter so the results could be queried, but I
>>>>>>>>>>>>> don't think it should be a default which would mean it's
>>>>>>>>>>>>> not reliable for the console to assume it's getting back
>>>>>>>>>>>>> JSON.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I've also thought about, haven't tested this and it may
>>>>>>>>>>>>> not work at all, creating a handler that uses websockets
>>>>>>>>>>>>> to send messages. I'm not sure how well this would work
>>>>>>>>>>>>> and it's possible it may not even work for bootstrap logging.
>>>>>>>>>>>>>
>>>>>>>>>>>>> With regards to audit logging, we're probably going to
>>>>>>>>>>>>> have to do something totally different from what we'll do
>>>>>>>>>>>>> in the logging subsystem since it doesn't use standard
>>>>>>>>>>>>> logging.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I guess the bottom line is what does the console want to
>>>>>>>>>>>>> see? Do you want to see all raw text log messages? Do you
>>>>>>>>>>>>> want all messages but in a format like JSON that you can
>>>>>>>>>>>>> query/filter? Do you really want only the last 10 error
>>>>>>>>>>>>> messages only? All or none of these might be possible, but
>>>>>>>>>>>>> I really need to understand the needs before I can explore
>>>>>>>>>>>>> more in depth what the best option would be.
>>>>>>>>>>>>>
>>>>>>>>>>>>> [1]: https://issues.jboss.org/browse/WFLY-280
>>>>>>>>>>>>> --
>>>>>>>>>>>>> James R. Perkins
>>>>>>>>>>>>> Red Hat JBoss Middleware
>>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> James R. Perkins
>>>>>>>>>>>> Red Hat JBoss Middleware
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>> wildfly-dev mailing list
>>>>>>>>>>>>
>>>>>>>>>>>> wildfly-dev at lists.jboss.org
>>>>>>>>>>>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> wildfly-dev mailing list
>>>>>>>>>>>
>>>>>>>>>>> wildfly-dev at lists.jboss.org
>>>>>>>>>>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>>>>>>>>>> _______________________________________________
>>>>>>>>>> wildfly-dev mailing list
>>>>>>>>>> wildfly-dev at lists.jboss.org
>>>>>>>>>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>>>>>>>> _______________________________________________
>>>>>>>> wildfly-dev mailing list
>>>>>>>> wildfly-dev at lists.jboss.org
>>>>>>>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>>>
>>
>> --
>> James R. Perkins
>> Red Hat JBoss Middleware
>> _______________________________________________
>> wildfly-dev mailing list
>> wildfly-dev at lists.jboss.org <mailto:wildfly-dev at lists.jboss.org>
>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>
--
James R. Perkins
Red Hat JBoss Middleware
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/wildfly-dev/attachments/20131024/98d2840b/attachment-0001.html
More information about the wildfly-dev
mailing list