Re: [wildfly-dev] Reading Logs from Web Console

Is it possible to read log files not stored under ${jboss.server.log.dir}? I'm asking because the audit log is per default stored under ${jboss.server.data.dir} --- Harald Pehl JBoss by Red Hat http://hpehl.info Am 09.10.2013 um 18:10 schrieb James R. Perkins <jperkins(a)redhat.com <mailto:jperkins@redhat.com>>: > In my opinion we shouldn't offer any kind of log parsing options. We > should simple deliver the raw log file and have the end user use > their own parsing. There is a JIRA [1] to allow custom formatters to > be used. This means an end user could use an XML formatter and it > could be easily parsed by them. Since format patterns are so > configurable and we're going to expose the ability to override the > formatter we need to just give them the raw file. > > I suppose saying all that contradicts my reasoning for using the > BufferedReader.readLine() and maybe we should be returning by number > of bytes/chars instead of by line. There seems to be no real good or > solid answer and it would be nice to hear from any users that have an > opinion on how they might use it. From the web console I'm going to > guess either approach, by line or by bytes, will work fine as it > might just construct one large file for download. If using it from a > console just to see the last 20 lines of the log file, a line-by-line > approach to me makes more sense. > > [1]: https://issues.jboss.org/browse/WFLY-1188 > > On 10/09/2013 05:09 AM, Ondrej Zizka wrote: >> How about storing the offsets of the records. >> Then the operation would read this "index" and instead of >> format-based parsing, it would simply cut the right part. >> Simple, fast in both write and read, and doesn't consume too much >> memory/storage. >> >> Ondra >> >> >> On 8.10.2013 18:57, James R. Perkins wrote: >>> I'm definitely not trying to parse any of the log message format. I am >>> using a BufferedReader.readLine() which I was a little iffy on, but >>> went >>> with "lines" as opposed to reading x bytes. Obviously from a >>> management >>> standpoint reading the number of bytes makes more sense as we're not >>> relying on any kind of normal line terminator being used. From a user >>> standpoint though I think it makes less sense and the number of lines >>> makes more sense. >>> >>> I suppose we could have two operations so the web console could use >>> the >>> one that reads bytes and then have one a simple tail like one that >>> reads >>> by line. >>> >>> On 10/08/2013 09:26 AM, David M. Lloyd wrote: >>>> I've said it before and I'll say it again. Using the format to >>>> "intelligently" parse the logs looks like a good idea. It is >>>> actually a >>>> bad idea. A bad idea that looks like a good idea, or maybe even a >>>> "good >>>> enough" idea. Maybe it looks like an "easy" idea. But it's bad. >>>> Bad, >>>> and also not good. It will result in bug reports that nobody will >>>> bother fixing or implementing because they will be reasonable yet >>>> unfeasible to fix. >>>> >>>> Here are some implementation options that I *won't* immediately kill: >>>> >>>> 1) Treat the files in the log directory as opaque things that the >>>> server >>>> does not pretend to have interior knowledge of - in other words, >>>> provide >>>> operations that work on the file as a whole (or by things which make >>>> sense at a file level, like line number or dumb string grep) which >>>> will >>>> work 100% of the time >>>> 2) Provide in-memory (object) or on-disk (structured binary) >>>> storage of >>>> log record objects with intelligent search capabilities which will >>>> work >>>> 100% of the time >>>> >>>> Here are some implementation options that I *will* immediately kill: >>>> >>>> 1) Try to use the format to figure out record boundaries >>>> 2) Try to use the format to figure out the date of the record >>>> 3) Try to use the format to extract exception information >>>> 4) Any and all other format-based ideas not mentioned here >>>> >>>> The common theme is - it has to work 100% of the time. No excuses. >>>> >>>> On 10/08/2013 10:55 AM, James R. Perkins wrote: >>>>> On 10/08/2013 03:40 AM, Kabir Khan wrote: >>>>>> Just a thought regarding your PR >>>>>> https://github.com/wildfly/wildfly/pull/5156 >>>>>> >>>>>> The thing that stands out to me is that you're able to read the >>>>>> first 100 lines, e.g. >>>>>> :read-log-file(name=server.log, lines=100, tail=false) >>>>>> or the last 100 lines >>>>>> :read-log-file(name=server.log, lines=100, tail=true) >>>>>> >>>>>> Would it also be beneficial to also be allowed to start >>>>>> somewhere in the middle, e.g >>>>>> :read-log-file(name=server.log, lines=100, start=10235) >>>>> There is an option skip that will allows something similar. The >>>>> following would read read the last 100 lines _after_ 50 lines are >>>>> skipped from the bottom. So in a 200 line this would read lines >>>>> 50 to >>>>> line 150. >>>>> :read-log-file(name=server.log, lines=100, skip=50, >>>>> tail=true) >>>>> >>>>> The following would read from line 50 to line 150 >>>>> :read-log-file(name=server.log, lines=100, skip=50, >>>>> tail=false) >>>>> >>>>> I am using a BufferedReader so if an odd line terminator is used >>>>> it will >>>>> likely not work so well. I considered allowing for the line >>>>> terminator >>>>> to be defined in the operation, but it would require analyzing >>>>> bytes for >>>>> the terminator and seemed like more work than it was worth. I would >>>>> guess the majority of users use the standard /n or /r/n terminators. >>>>>> or (19:38:01 today) >>>>>> :read-log-file(name=server.log, lines=100, >>>>>> start-time=20131008-10:38:01) >>>>> The date would be rather difficult. We're not parsing the log >>>>> strings >>>>> just reading the raw text and sending it back. >>>>>> I'm not saying it MUST be like that but it might be worth >>>>>> discussing >>>>>> >>>>>> On 25 Sep 2013, at 16:36, Ondrej Zizka wrote: >>>>>> >>>>>>> Right, by "current" I meant to read the logging setup and if >>>>>>> there's >>>>>>> just one file appender, take it as default. Or, if that would be >>>>>>> inappropriate mixing of abstraction layers, just "server.log". >>>>>>> >>>>>>> >>>>>>> On 25.9.2013 12:06, Kabir Khan wrote: >>>>>>>> Current log could be the server.log in a standard setup. If >>>>>>>> someone has done something more advanced in their logging >>>>>>>> setup, the name becomes necessary >>>>>>>> On 25 Sep 2013, at 08:34, Jaikiran Pai wrote: >>>>>>>> >>>>>>>>> On Wednesday 25 September 2013 09:25 AM, Ondrej Zizka wrote: >>>>>>>>>> 1) Could it have a "read-log-file()" without name= >>>>>>>>>> specified, which would read the "current" log file? >>>>>>>>> Given the way logger categories and appenders/handlers >>>>>>>>> interact, within a logging framework, I don't think there's >>>>>>>>> any notion of "current log file". It's a very valid scenario >>>>>>>>> where a single logging category can be backed by different >>>>>>>>> appenders (some of them file appenders) with different >>>>>>>>> attributes and each such appender writing out to a different >>>>>>>>> file. So having a name of the log file you want to view, >>>>>>>>> becomes necessary. >>>>>>>>> >>>>>>>>> -Jaikiran >>>>>>>>> >>>>>>>>>> 2) Regarding security - what, besides logs, do we expect to >>>>>>>>>> be in the log dir? Could the admin block it by setting >>>>>>>>>> write-only rights? >>>>>>>>>> >>>>>>>>>> Ondra >>>>>>>>>> >>>>>>>>>> On 25.9.2013 02:40, James R. Perkins wrote: >>>>>>>>>>> I'm replying to this old thread to reopen this conversation >>>>>>>>>>> about reading log files. I've complete some work [1] on >>>>>>>>>>> reading log files via an operation. This is not exactly >>>>>>>>>>> like the JIRA suggests where it would only read the last 10 >>>>>>>>>>> error messages. All this change allows is the raw contents >>>>>>>>>>> of the file to be read. The idea is this could be used to >>>>>>>>>>> read the entire contents of the log file as a whole, or in >>>>>>>>>>> chunks. >>>>>>>>>>> >>>>>>>>>>> What I've done is added two new operations list-log-files >>>>>>>>>>> and read-log-file. >>>>>>>>>>> >>>>>>>>>>> The list-log-files simply lists all files in the >>>>>>>>>>> jboss.server.log.dir. This may or may not be a good idea >>>>>>>>>>> really. I can see some potential security risks here mainly >>>>>>>>>>> just seeing files that may contain sensitive data. One way >>>>>>>>>>> I've thought of to get around that is read the logging >>>>>>>>>>> subsystem model and only show files from known types like >>>>>>>>>>> the file-handlers. The main issue with that is there is no >>>>>>>>>>> good way to get this to work for custom-handlers. >>>>>>>>>>> >>>>>>>>>>> The read-log-file simple does what it says and reads the >>>>>>>>>>> contents of a log file line by line. Reading line by line >>>>>>>>>>> should work for the most part unless the an non-standard >>>>>>>>>>> line delimiter is used. There are 5 options for this option; >>>>>>>>>>> • name (required): the name of the log file to read >>>>>>>>>>> • encoding: the encoding for the log file >>>>>>>>>>> • lines: the number of lines to read, defaults to 10 >>>>>>>>>>> • skip: the number of lines to skip before adding the >>>>>>>>>>> results >>>>>>>>>>> • tail: true to read from the bottom up, default is true >>>>>>>>>>> The result of this is just a list of lines with the \n or >>>>>>>>>>> \r\n stripped. Just to clarify too a line means a line in >>>>>>>>>>> the file, not a log record e.g. stack traces are generally >>>>>>>>>>> composed of multiple lines. >>>>>>>>>>> >>>>>>>>>>> So this begs the question, will this work for what we want? >>>>>>>>>>> What concerns does anyone else have? >>>>>>>>>>> >>>>>>>>>>> I have not yet submitted a PR yet as I wanted to get some >>>>>>>>>>> feedback before we bake it in. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> [1]: https://github.com/jamezp/wildfly/compare/WFLY-280-read >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On 08/14/2013 10:03 AM, James R. Perkins wrote: >>>>>>>>>>>> I had posted this to another list, but this is a more >>>>>>>>>>>> appropriate place for it. I think there needs to be a >>>>>>>>>>>> general discussion around this as it's been mentioned, at >>>>>>>>>>>> least to me, a few times here and there and I know Heiko >>>>>>>>>>>> raised the issue some time a go now. >>>>>>>>>>>> >>>>>>>>>>>> The original JIRA, WFLY-280[1], is to display the last 10 >>>>>>>>>>>> error messages only. To be honest I wouldn't find that >>>>>>>>>>>> very useful. To me if I'm looking for logs I want to see >>>>>>>>>>>> all logs, but that's not always so easy. Like the >>>>>>>>>>>> syslog-handler which doesn't log to a file so there is no >>>>>>>>>>>> way to read those messages back. >>>>>>>>>>>> >>>>>>>>>>>> The current plan for the last 10 error messages is we >>>>>>>>>>>> store messages in a queue that can be accessed via an >>>>>>>>>>>> operation. This works fine until the error message you're >>>>>>>>>>>> interested in is 11 or you want to see warning messages. >>>>>>>>>>>> >>>>>>>>>>>> Another option I had come up with is reading back the >>>>>>>>>>>> contents of the file, for example the server.log. This >>>>>>>>>>>> could be problematic too in that there is no way to filter >>>>>>>>>>>> information like only see error messages or only see >>>>>>>>>>>> warning messages. To solve this I have considered creating >>>>>>>>>>>> a JSON formatter so the results could be queried, but I >>>>>>>>>>>> don't think it should be a default which would mean it's >>>>>>>>>>>> not reliable for the console to assume it's getting back >>>>>>>>>>>> JSON. >>>>>>>>>>>> >>>>>>>>>>>> I've also thought about, haven't tested this and it may >>>>>>>>>>>> not work at all, creating a handler that uses websockets >>>>>>>>>>>> to send messages. I'm not sure how well this would work >>>>>>>>>>>> and it's possible it may not even work for bootstrap logging. >>>>>>>>>>>> >>>>>>>>>>>> With regards to audit logging, we're probably going to >>>>>>>>>>>> have to do something totally different from what we'll do >>>>>>>>>>>> in the logging subsystem since it doesn't use standard >>>>>>>>>>>> logging. >>>>>>>>>>>> >>>>>>>>>>>> I guess the bottom line is what does the console want to >>>>>>>>>>>> see? Do you want to see all raw text log messages? Do you >>>>>>>>>>>> want all messages but in a format like JSON that you can >>>>>>>>>>>> query/filter? Do you really want only the last 10 error >>>>>>>>>>>> messages only? All or none of these might be possible, but >>>>>>>>>>>> I really need to understand the needs before I can explore >>>>>>>>>>>> more in depth what the best option would be. >>>>>>>>>>>> >>>>>>>>>>>> [1]: https://issues.jboss.org/browse/WFLY-280 >>>>>>>>>>>> -- >>>>>>>>>>>> James R. Perkins >>>>>>>>>>>> Red Hat JBoss Middleware >>>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> James R. Perkins >>>>>>>>>>> Red Hat JBoss Middleware >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> wildfly-dev mailing list >>>>>>>>>>> >>>>>>>>>>> wildfly-dev(a)lists.jboss.org >>>>>>>>>>> https://lists.jboss.org/mailman/listinfo/wildfly-dev >>>>>>>>>> _______________________________________________ >>>>>>>>>> wildfly-dev mailing list >>>>>>>>>> >>>>>>>>>> wildfly-dev(a)lists.jboss.org >>>>>>>>>> https://lists.jboss.org/mailman/listinfo/wildfly-dev >>>>>>>>> _______________________________________________ >>>>>>>>> wildfly-dev mailing list >>>>>>>>> wildfly-dev(a)lists.jboss.org >>>>>>>>> https://lists.jboss.org/mailman/listinfo/wildfly-dev >>>>>>> _______________________________________________ >>>>>>> wildfly-dev mailing list >>>>>>> wildfly-dev(a)lists.jboss.org >>>>>>> https://lists.jboss.org/mailman/listinfo/wildfly-dev >> > > -- > James R. Perkins > Red Hat JBoss Middleware > _______________________________________________ > wildfly-dev mailing list > wildfly-dev(a)lists.jboss.org <mailto:wildfly-dev@lists.jboss.org> > https://lists.jboss.org/mailman/listinfo/wildfly-dev