Re: [wildfly-dev] Reading Logs from Web Console

In my opinion we shouldn't offer any kind of log parsing options. We should simple deliver the raw log file and have the end user use their own parsing. There is a JIRA [1] to allow custom formatters to be used. This means an end user could use an XML formatter and it could be easily parsed by them. Since format patterns are so configurable and we're going to expose the ability to override the formatter we need to just give them the raw file. I suppose saying all that contradicts my reasoning for using the BufferedReader.readLine() and maybe we should be returning by number of bytes/chars instead of by line. There seems to be no real good or solid answer and it would be nice to hear from any users that have an opinion on how they might use it. From the web console I'm going to guess either approach, by line or by bytes, will work fine as it might just construct one large file for download. If using it from a console just to see the last 20 lines of the log file, a line-by-line approach to me makes more sense. [1]: https://issues.jboss.org/browse/WFLY-1188 On 10/09/2013 05:09 AM, Ondrej Zizka wrote: > How about storing the offsets of the records. > Then the operation would read this "index" and instead of format-based parsing, it would simply cut the right part. > Simple, fast in both write and read, and doesn't consume too much memory/storage. > > Ondra > > > On 8.10.2013 18:57, James R. Perkins wrote: >> I'm definitely not trying to parse any of the log message format. I am >> using a BufferedReader.readLine() which I was a little iffy on, but went >> with "lines" as opposed to reading x bytes. Obviously from a management >> standpoint reading the number of bytes makes more sense as we're not >> relying on any kind of normal line terminator being used. From a user >> standpoint though I think it makes less sense and the number of lines >> makes more sense. >> >> I suppose we could have two operations so the web console could use the >> one that reads bytes and then have one a simple tail like one that reads >> by line. >> >> On 10/08/2013 09:26 AM, David M. Lloyd wrote: >>> I've said it before and I'll say it again. Using the format to >>> "intelligently" parse the logs looks like a good idea. It is actually a >>> bad idea. A bad idea that looks like a good idea, or maybe even a "good >>> enough" idea. Maybe it looks like an "easy" idea. But it's bad. Bad, >>> and also not good. It will result in bug reports that nobody will >>> bother fixing or implementing because they will be reasonable yet >>> unfeasible to fix. >>> >>> Here are some implementation options that I *won't* immediately kill: >>> >>> 1) Treat the files in the log directory as opaque things that the server >>> does not pretend to have interior knowledge of - in other words, provide >>> operations that work on the file as a whole (or by things which make >>> sense at a file level, like line number or dumb string grep) which will >>> work 100% of the time >>> 2) Provide in-memory (object) or on-disk (structured binary) storage of >>> log record objects with intelligent search capabilities which will work >>> 100% of the time >>> >>> Here are some implementation options that I *will* immediately kill: >>> >>> 1) Try to use the format to figure out record boundaries >>> 2) Try to use the format to figure out the date of the record >>> 3) Try to use the format to extract exception information >>> 4) Any and all other format-based ideas not mentioned here >>> >>> The common theme is - it has to work 100% of the time. No excuses. >>> >>> On 10/08/2013 10:55 AM, James R. Perkins wrote: >>>> On 10/08/2013 03:40 AM, Kabir Khan wrote: >>>>> Just a thought regarding your PR https://github.com/wildfly/wildfly/pull/5156 >>>>> >>>>> The thing that stands out to me is that you're able to read the first 100 lines, e.g. >>>>> :read-log-file(name=server.log, lines=100, tail=false) >>>>> or the last 100 lines >>>>> :read-log-file(name=server.log, lines=100, tail=true) >>>>> >>>>> Would it also be beneficial to also be allowed to start somewhere in the middle, e.g >>>>> :read-log-file(name=server.log, lines=100, start=10235) >>>> There is an option skip that will allows something similar. The >>>> following would read read the last 100 lines _after_ 50 lines are >>>> skipped from the bottom. So in a 200 line this would read lines 50 to >>>> line 150. >>>> :read-log-file(name=server.log, lines=100, skip=50, tail=true) >>>> >>>> The following would read from line 50 to line 150 >>>> :read-log-file(name=server.log, lines=100, skip=50, tail=false) >>>> >>>> I am using a BufferedReader so if an odd line terminator is used it will >>>> likely not work so well. I considered allowing for the line terminator >>>> to be defined in the operation, but it would require analyzing bytes for >>>> the terminator and seemed like more work than it was worth. I would >>>> guess the majority of users use the standard /n or /r/n terminators. >>>>> or (19:38:01 today) >>>>> :read-log-file(name=server.log, lines=100, start-time=20131008-10:38:01) >>>> The date would be rather difficult. We're not parsing the log strings >>>> just reading the raw text and sending it back. >>>>> I'm not saying it MUST be like that but it might be worth discussing >>>>> >>>>> On 25 Sep 2013, at 16:36, Ondrej Zizka wrote: >>>>> >>>>>> Right, by "current" I meant to read the logging setup and if there's >>>>>> just one file appender, take it as default. Or, if that would be >>>>>> inappropriate mixing of abstraction layers, just "server.log". >>>>>> >>>>>> >>>>>> On 25.9.2013 12:06, Kabir Khan wrote: >>>>>>> Current log could be the server.log in a standard setup. If someone has done something more advanced in their logging setup, the name becomes necessary >>>>>>> On 25 Sep 2013, at 08:34, Jaikiran Pai wrote: >>>>>>> >>>>>>>> On Wednesday 25 September 2013 09:25 AM, Ondrej Zizka wrote: >>>>>>>>> 1) Could it have a "read-log-file()" without name= specified, which would read the "current" log file? >>>>>>>> Given the way logger categories and appenders/handlers interact, within a logging framework, I don't think there's any notion of "current log file". It's a very valid scenario where a single logging category can be backed by different appenders (some of them file appenders) with different attributes and each such appender writing out to a different file. So having a name of the log file you want to view, becomes necessary. >>>>>>>> >>>>>>>> -Jaikiran >>>>>>>> >>>>>>>>> 2) Regarding security - what, besides logs, do we expect to be in the log dir? Could the admin block it by setting write-only rights? >>>>>>>>> >>>>>>>>> Ondra >>>>>>>>> >>>>>>>>> On 25.9.2013 02:40, James R. Perkins wrote: >>>>>>>>>> I'm replying to this old thread to reopen this conversation about reading log files. I've complete some work [1] on reading log files via an operation. This is not exactly like the JIRA suggests where it would only read the last 10 error messages. All this change allows is the raw contents of the file to be read. The idea is this could be used to read the entire contents of the log file as a whole, or in chunks. >>>>>>>>>> >>>>>>>>>> What I've done is added two new operations list-log-files and read-log-file. >>>>>>>>>> >>>>>>>>>> The list-log-files simply lists all files in the jboss.server.log.dir. This may or may not be a good idea really. I can see some potential security risks here mainly just seeing files that may contain sensitive data. One way I've thought of to get around that is read the logging subsystem model and only show files from known types like the file-handlers. The main issue with that is there is no good way to get this to work for custom-handlers. >>>>>>>>>> >>>>>>>>>> The read-log-file simple does what it says and reads the contents of a log file line by line. Reading line by line should work for the most part unless the an non-standard line delimiter is used. There are 5 options for this option; >>>>>>>>>> • name (required): the name of the log file to read >>>>>>>>>> • encoding: the encoding for the log file >>>>>>>>>> • lines: the number of lines to read, defaults to 10 >>>>>>>>>> • skip: the number of lines to skip before adding the results >>>>>>>>>> • tail: true to read from the bottom up, default is true >>>>>>>>>> The result of this is just a list of lines with the \n or \r\n stripped. Just to clarify too a line means a line in the file, not a log record e.g. stack traces are generally composed of multiple lines. >>>>>>>>>> >>>>>>>>>> So this begs the question, will this work for what we want? What concerns does anyone else have? >>>>>>>>>> >>>>>>>>>> I have not yet submitted a PR yet as I wanted to get some feedback before we bake it in. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> [1]: https://github.com/jamezp/wildfly/compare/WFLY-280-read >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On 08/14/2013 10:03 AM, James R. Perkins wrote: >>>>>>>>>>> I had posted this to another list, but this is a more appropriate place for it. I think there needs to be a general discussion around this as it's been mentioned, at least to me, a few times here and there and I know Heiko raised the issue some time a go now. >>>>>>>>>>> >>>>>>>>>>> The original JIRA, WFLY-280[1], is to display the last 10 error messages only. To be honest I wouldn't find that very useful. To me if I'm looking for logs I want to see all logs, but that's not always so easy. Like the syslog-handler which doesn't log to a file so there is no way to read those messages back. >>>>>>>>>>> >>>>>>>>>>> The current plan for the last 10 error messages is we store messages in a queue that can be accessed via an operation. This works fine until the error message you're interested in is 11 or you want to see warning messages. >>>>>>>>>>> >>>>>>>>>>> Another option I had come up with is reading back the contents of the file, for example the server.log. This could be problematic too in that there is no way to filter information like only see error messages or only see warning messages. To solve this I have considered creating a JSON formatter so the results could be queried, but I don't think it should be a default which would mean it's not reliable for the console to assume it's getting back JSON. >>>>>>>>>>> >>>>>>>>>>> I've also thought about, haven't tested this and it may not work at all, creating a handler that uses websockets to send messages. I'm not sure how well this would work and it's possible it may not even work for bootstrap logging. >>>>>>>>>>> >>>>>>>>>>> With regards to audit logging, we're probably going to have to do something totally different from what we'll do in the logging subsystem since it doesn't use standard logging. >>>>>>>>>>> >>>>>>>>>>> I guess the bottom line is what does the console want to see? Do you want to see all raw text log messages? Do you want all messages but in a format like JSON that you can query/filter? Do you really want only the last 10 error messages only? All or none of these might be possible, but I really need to understand the needs before I can explore more in depth what the best option would be. >>>>>>>>>>> >>>>>>>>>>> [1]: https://issues.jboss.org/browse/WFLY-280 >>>>>>>>>>> -- >>>>>>>>>>> James R. Perkins >>>>>>>>>>> Red Hat JBoss Middleware >>>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> James R. Perkins >>>>>>>>>> Red Hat JBoss Middleware >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> wildfly-dev mailing list >>>>>>>>>> >>>>>>>>>> wildfly-dev(a)lists.jboss.org >>>>>>>>>> https://lists.jboss.org/mailman/listinfo/wildfly-dev >>>>>>>>> _______________________________________________ >>>>>>>>> wildfly-dev mailing list >>>>>>>>> >>>>>>>>> wildfly-dev(a)lists.jboss.org >>>>>>>>> https://lists.jboss.org/mailman/listinfo/wildfly-dev >>>>>>>> _______________________________________________ >>>>>>>> wildfly-dev mailing list >>>>>>>> wildfly-dev(a)lists.jboss.org >>>>>>>> https://lists.jboss.org/mailman/listinfo/wildfly-dev >>>>>> _______________________________________________ >>>>>> wildfly-dev mailing list >>>>>> wildfly-dev(a)lists.jboss.org >>>>>> https://lists.jboss.org/mailman/listinfo/wildfly-dev > -- James R. Perkins Red Hat JBoss Middleware _______________________________________________ wildfly-dev mailing list wildfly-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/wildfly-dev