Re: [wildfly-dev] Reading Logs from Web Console

How about storing the offsets of the records. Then the operation would read this "index" and instead of format-based parsing, it would simply cut the right part. Simple, fast in both write and read, and doesn't consume too much memory/storage. Ondra On 8.10.2013 18:57, James R. Perkins wrote: > I'm definitely not trying to parse any of the log message format. I am > using a BufferedReader.readLine() which I was a little iffy on, but went > with "lines" as opposed to reading x bytes. Obviously from a management > standpoint reading the number of bytes makes more sense as we're not > relying on any kind of normal line terminator being used. From a user > standpoint though I think it makes less sense and the number of lines > makes more sense. > > I suppose we could have two operations so the web console could use the > one that reads bytes and then have one a simple tail like one that reads > by line. > > On 10/08/2013 09:26 AM, David M. Lloyd wrote: >> I've said it before and I'll say it again. Using the format to >> "intelligently" parse the logs looks like a good idea. It is >> actually a >> bad idea. A bad idea that looks like a good idea, or maybe even a >> "good >> enough" idea. Maybe it looks like an "easy" idea. But it's bad. Bad, >> and also not good. It will result in bug reports that nobody will >> bother fixing or implementing because they will be reasonable yet >> unfeasible to fix. >> >> Here are some implementation options that I *won't* immediately kill: >> >> 1) Treat the files in the log directory as opaque things that the >> server >> does not pretend to have interior knowledge of - in other words, >> provide >> operations that work on the file as a whole (or by things which make >> sense at a file level, like line number or dumb string grep) which will >> work 100% of the time >> 2) Provide in-memory (object) or on-disk (structured binary) storage of >> log record objects with intelligent search capabilities which will work >> 100% of the time >> >> Here are some implementation options that I *will* immediately kill: >> >> 1) Try to use the format to figure out record boundaries >> 2) Try to use the format to figure out the date of the record >> 3) Try to use the format to extract exception information >> 4) Any and all other format-based ideas not mentioned here >> >> The common theme is - it has to work 100% of the time. No excuses. >> >> On 10/08/2013 10:55 AM, James R. Perkins wrote: >>> On 10/08/2013 03:40 AM, Kabir Khan wrote: >>>> Just a thought regarding your PR >>>> https://github.com/wildfly/wildfly/pull/5156 >>>> >>>> The thing that stands out to me is that you're able to read the >>>> first 100 lines, e.g. >>>> :read-log-file(name=server.log, lines=100, tail=false) >>>> or the last 100 lines >>>> :read-log-file(name=server.log, lines=100, tail=true) >>>> >>>> Would it also be beneficial to also be allowed to start somewhere >>>> in the middle, e.g >>>> :read-log-file(name=server.log, lines=100, start=10235) >>> There is an option skip that will allows something similar. The >>> following would read read the last 100 lines _after_ 50 lines are >>> skipped from the bottom. So in a 200 line this would read lines 50 to >>> line 150. >>> :read-log-file(name=server.log, lines=100, skip=50, tail=true) >>> >>> The following would read from line 50 to line 150 >>> :read-log-file(name=server.log, lines=100, skip=50, >>> tail=false) >>> >>> I am using a BufferedReader so if an odd line terminator is used it >>> will >>> likely not work so well. I considered allowing for the line terminator >>> to be defined in the operation, but it would require analyzing >>> bytes for >>> the terminator and seemed like more work than it was worth. I would >>> guess the majority of users use the standard /n or /r/n terminators. >>>> or (19:38:01 today) >>>> :read-log-file(name=server.log, lines=100, >>>> start-time=20131008-10:38:01) >>> The date would be rather difficult. We're not parsing the log strings >>> just reading the raw text and sending it back. >>>> I'm not saying it MUST be like that but it might be worth discussing >>>> >>>> On 25 Sep 2013, at 16:36, Ondrej Zizka wrote: >>>> >>>>> Right, by "current" I meant to read the logging setup and if there's >>>>> just one file appender, take it as default. Or, if that would be >>>>> inappropriate mixing of abstraction layers, just "server.log". >>>>> >>>>> >>>>> On 25.9.2013 12:06, Kabir Khan wrote: >>>>>> Current log could be the server.log in a standard setup. If >>>>>> someone has done something more advanced in their logging setup, >>>>>> the name becomes necessary >>>>>> On 25 Sep 2013, at 08:34, Jaikiran Pai wrote: >>>>>> >>>>>>> On Wednesday 25 September 2013 09:25 AM, Ondrej Zizka wrote: >>>>>>>> 1) Could it have a "read-log-file()" without name= specified, >>>>>>>> which would read the "current" log file? >>>>>>> Given the way logger categories and appenders/handlers >>>>>>> interact, within a logging framework, I don't think there's any >>>>>>> notion of "current log file". It's a very valid scenario where >>>>>>> a single logging category can be backed by different appenders >>>>>>> (some of them file appenders) with different attributes and >>>>>>> each such appender writing out to a different file. So having a >>>>>>> name of the log file you want to view, becomes necessary. >>>>>>> >>>>>>> -Jaikiran >>>>>>> >>>>>>>> 2) Regarding security - what, besides logs, do we expect to be >>>>>>>> in the log dir? Could the admin block it by setting write-only >>>>>>>> rights? >>>>>>>> >>>>>>>> Ondra >>>>>>>> >>>>>>>> On 25.9.2013 02:40, James R. Perkins wrote: >>>>>>>>> I'm replying to this old thread to reopen this conversation >>>>>>>>> about reading log files. I've complete some work [1] on >>>>>>>>> reading log files via an operation. This is not exactly like >>>>>>>>> the JIRA suggests where it would only read the last 10 error >>>>>>>>> messages. All this change allows is the raw contents of the >>>>>>>>> file to be read. The idea is this could be used to read the >>>>>>>>> entire contents of the log file as a whole, or in chunks. >>>>>>>>> >>>>>>>>> What I've done is added two new operations list-log-files and >>>>>>>>> read-log-file. >>>>>>>>> >>>>>>>>> The list-log-files simply lists all files in the >>>>>>>>> jboss.server.log.dir. This may or may not be a good idea >>>>>>>>> really. I can see some potential security risks here mainly >>>>>>>>> just seeing files that may contain sensitive data. One way >>>>>>>>> I've thought of to get around that is read the logging >>>>>>>>> subsystem model and only show files from known types like the >>>>>>>>> file-handlers. The main issue with that is there is no good >>>>>>>>> way to get this to work for custom-handlers. >>>>>>>>> >>>>>>>>> The read-log-file simple does what it says and reads the >>>>>>>>> contents of a log file line by line. Reading line by line >>>>>>>>> should work for the most part unless the an non-standard line >>>>>>>>> delimiter is used. There are 5 options for this option; >>>>>>>>> • name (required): the name of the log file to read >>>>>>>>> • encoding: the encoding for the log file >>>>>>>>> • lines: the number of lines to read, defaults to 10 >>>>>>>>> • skip: the number of lines to skip before adding the >>>>>>>>> results >>>>>>>>> • tail: true to read from the bottom up, default is true >>>>>>>>> The result of this is just a list of lines with the \n or >>>>>>>>> \r\n stripped. Just to clarify too a line means a line in the >>>>>>>>> file, not a log record e.g. stack traces are generally >>>>>>>>> composed of multiple lines. >>>>>>>>> >>>>>>>>> So this begs the question, will this work for what we want? >>>>>>>>> What concerns does anyone else have? >>>>>>>>> >>>>>>>>> I have not yet submitted a PR yet as I wanted to get some >>>>>>>>> feedback before we bake it in. >>>>>>>>> >>>>>>>>> >>>>>>>>> [1]: https://github.com/jamezp/wildfly/compare/WFLY-280-read >>>>>>>>> >>>>>>>>> >>>>>>>>> On 08/14/2013 10:03 AM, James R. Perkins wrote: >>>>>>>>>> I had posted this to another list, but this is a more >>>>>>>>>> appropriate place for it. I think there needs to be a >>>>>>>>>> general discussion around this as it's been mentioned, at >>>>>>>>>> least to me, a few times here and there and I know Heiko >>>>>>>>>> raised the issue some time a go now. >>>>>>>>>> >>>>>>>>>> The original JIRA, WFLY-280[1], is to display the last 10 >>>>>>>>>> error messages only. To be honest I wouldn't find that very >>>>>>>>>> useful. To me if I'm looking for logs I want to see all >>>>>>>>>> logs, but that's not always so easy. Like the syslog-handler >>>>>>>>>> which doesn't log to a file so there is no way to read those >>>>>>>>>> messages back. >>>>>>>>>> >>>>>>>>>> The current plan for the last 10 error messages is we store >>>>>>>>>> messages in a queue that can be accessed via an operation. >>>>>>>>>> This works fine until the error message you're interested in >>>>>>>>>> is 11 or you want to see warning messages. >>>>>>>>>> >>>>>>>>>> Another option I had come up with is reading back the >>>>>>>>>> contents of the file, for example the server.log. This could >>>>>>>>>> be problematic too in that there is no way to filter >>>>>>>>>> information like only see error messages or only see warning >>>>>>>>>> messages. To solve this I have considered creating a JSON >>>>>>>>>> formatter so the results could be queried, but I don't think >>>>>>>>>> it should be a default which would mean it's not reliable >>>>>>>>>> for the console to assume it's getting back JSON. >>>>>>>>>> >>>>>>>>>> I've also thought about, haven't tested this and it may not >>>>>>>>>> work at all, creating a handler that uses websockets to send >>>>>>>>>> messages. I'm not sure how well this would work and it's >>>>>>>>>> possible it may not even work for bootstrap logging. >>>>>>>>>> >>>>>>>>>> With regards to audit logging, we're probably going to have >>>>>>>>>> to do something totally different from what we'll do in the >>>>>>>>>> logging subsystem since it doesn't use standard logging. >>>>>>>>>> >>>>>>>>>> I guess the bottom line is what does the console want to >>>>>>>>>> see? Do you want to see all raw text log messages? Do you >>>>>>>>>> want all messages but in a format like JSON that you can >>>>>>>>>> query/filter? Do you really want only the last 10 error >>>>>>>>>> messages only? All or none of these might be possible, but I >>>>>>>>>> really need to understand the needs before I can explore >>>>>>>>>> more in depth what the best option would be. >>>>>>>>>> >>>>>>>>>> [1]: https://issues.jboss.org/browse/WFLY-280 >>>>>>>>>> -- >>>>>>>>>> James R. Perkins >>>>>>>>>> Red Hat JBoss Middleware >>>>>>>>>> >>>>>>>>> -- >>>>>>>>> James R. Perkins >>>>>>>>> Red Hat JBoss Middleware >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> wildfly-dev mailing list >>>>>>>>> >>>>>>>>> wildfly-dev(a)lists.jboss.org >>>>>>>>> https://lists.jboss.org/mailman/listinfo/wildfly-dev >>>>>>>> _______________________________________________ >>>>>>>> wildfly-dev mailing list >>>>>>>> >>>>>>>> wildfly-dev(a)lists.jboss.org >>>>>>>> https://lists.jboss.org/mailman/listinfo/wildfly-dev >>>>>>> _______________________________________________ >>>>>>> wildfly-dev mailing list >>>>>>> wildfly-dev(a)lists.jboss.org >>>>>>> https://lists.jboss.org/mailman/listinfo/wildfly-dev >>>>> _______________________________________________ >>>>> wildfly-dev mailing list >>>>> wildfly-dev(a)lists.jboss.org >>>>> https://lists.jboss.org/mailman/listinfo/wildfly-dev