I had posted this to another list, but this is a more appropriate place for it. I think there needs to be a general discussion around this as it's been mentioned, at least to me, a few times here and there and I know Heiko raised the issue some time a go now. The original JIRA, WFLY-280[1], is to display the last 10 error messages only. To be honest I wouldn't find that very useful. To me if I'm looking for logs I want to see all logs, but that's not always so easy. Like the syslog-handler which doesn't log to a file so there is no way to read those messages back. The current plan for the last 10 error messages is we store messages in a queue that can be accessed via an operation. This works fine until the error message you're interested in is 11 or you want to see warning messages. Another option I had come up with is reading back the contents of the file, for example the server.log. This could be problematic too in that there is no way to filter information like only see error messages or only see warning messages. To solve this I have considered creating a JSON formatter so the results could be queried, but I don't think it should be a default which would mean it's not reliable for the console to assume it's getting back JSON. I've also thought about, haven't tested this and it may not work at all, creating a handler that uses websockets to send messages. I'm not sure how well this would work and it's possible it may not even work for bootstrap logging. With regards to audit logging, we're probably going to have to do something totally different from what we'll do in the logging subsystem since it doesn't use standard logging. I guess the bottom line is what does the console want to see? Do you want to see all raw text log messages? Do you want all messages but in a format like JSON that you can query/filter? Do you really want only the last 10 error messages only? All or none of these might be possible, but I really need to understand the needs before I can explore more in depth what the best option would be. [1]: https://issues.jboss.org/browse/WFLY-280 -- James R. Perkins Red Hat JBoss Middleware _______________________________________________ wildfly-dev mailing list wildfly-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/wildfly-dev

That's my thinking too. The only complaint I've had about that solution is that it can't be filtered since it's just raw text. I had a working example operation that just took a file name and the number of bytes to read. To me this is the best solution, but it wouldn't have access to anything in a syslog or the console. Though that's okay IMO since they likely have other viewers for those. I'll resurrect that example. It shouldn't take all that long. One thing to note is it will only allow files to be read that are in the jboss.server.log.dir (and possibly only files that end in *.log?). So in general I agree here, I think this is the best, the least invasive approach and the lowest performance hit. On 08/14/2013 10:36 AM, Jason Greene wrote: > IMO the best solution is a management operation that simply displays portions of the log file. The dependency on the log file is IMO not a problem because we support multiple appenders. > > On Aug 14, 2013, at 12:03 PM, James R. Perkins <jperkins(a)redhat.com&gt; wrote: > >> I had posted this to another list, but this is a more appropriate place for it. I think there needs to be a general discussion around this as it's been mentioned, at least to me, a few times here and there and I know Heiko raised the issue some time a go now. >> >> The original JIRA, WFLY-280[1], is to display the last 10 error messages only. To be honest I wouldn't find that very useful. To me if I'm looking for logs I want to see all logs, but that's not always so easy. Like the syslog-handler which doesn't log to a file so there is no way to read those messages back. >> >> The current plan for the last 10 error messages is we store messages in a queue that can be accessed via an operation. This works fine until the error message you're interested in is 11 or you want to see warning messages. >> >> Another option I had come up with is reading back the contents of the file, for example the server.log. This could be problematic too in that there is no way to filter information like only see error messages or only see warning messages. To solve this I have considered creating a JSON formatter so the results could be queried, but I don't think it should be a default which would mean it's not reliable for the console to assume it's getting back JSON. >> >> I've also thought about, haven't tested this and it may not work at all, creating a handler that uses websockets to send messages. I'm not sure how well this would work and it's possible it may not even work for bootstrap logging. >> >> With regards to audit logging, we're probably going to have to do something totally different from what we'll do in the logging subsystem since it doesn't use standard logging. >> >> I guess the bottom line is what does the console want to see? Do you want to see all raw text log messages? Do you want all messages but in a format like JSON that you can query/filter? Do you really want only the last 10 error messages only? All or none of these might be possible, but I really need to understand the needs before I can explore more in depth what the best option would be. >> >> [1]: https://issues.jboss.org/browse/WFLY-280 >> -- >> James R. Perkins >> Red Hat JBoss Middleware >> >> _______________________________________________ >> wildfly-dev mailing list >> wildfly-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/wildfly-dev > -- > Jason T. Greene > WildFly Lead / JBoss EAP Platform Architect > JBoss, a division of Red Hat > -- James R. Perkins Red Hat JBoss Middleware

The problem with filtering comes in parsing the log file. While I have had some success doing this with I tool I was playing with for Jesper, once you get a stack trace you have to start making guesses. I tend to agree with dmlloyd in that log files tend to be a one way write. You can do some parsing with a best guess, but well we all know what happens when we start assuming and guessing :) That said I do like the challenge of trying to parse log files. I did start a project to do it in my spare time I just haven't found time to put into working on it lately. On 08/14/2013 11:12 AM, Jason Greene wrote: > We could technically support server side filtering. We simply pattern search the file. We could also support client side filtering. We could for example utilize browser offline storage. The big issue is just that there are efficiency limits to the size of the file since it's not an indexed store. That could be improved by doing a real indexed data store (e.g. something like bdb). In the future I could picture an SPI that logging backends could implement for the purpose of searching. Although I think we will find that simply searching the normal text log file is good enough. > > On Aug 14, 2013, at 12:55 PM, James R. Perkins <jperkins(a)redhat.com&gt; wrote: > >> That's my thinking too. The only complaint I've had about that solution is that it can't be filtered since it's just raw text. I had a working example operation that just took a file name and the number of bytes to read. To me this is the best solution, but it wouldn't have access to anything in a syslog or the console. Though that's okay IMO since they likely have other viewers for those. >> >> I'll resurrect that example. It shouldn't take all that long. One thing to note is it will only allow files to be read that are in the jboss.server.log.dir (and possibly only files that end in *.log?). >> >> So in general I agree here, I think this is the best, the least invasive approach and the lowest performance hit. >> >> On 08/14/2013 10:36 AM, Jason Greene wrote: >>> IMO the best solution is a management operation that simply displays portions of the log file. The dependency on the log file is IMO not a problem because we support multiple appenders. >>> >>> On Aug 14, 2013, at 12:03 PM, James R. Perkins <jperkins(a)redhat.com&gt; wrote: >>> >>>> I had posted this to another list, but this is a more appropriate place for it. I think there needs to be a general discussion around this as it's been mentioned, at least to me, a few times here and there and I know Heiko raised the issue some time a go now. >>>> >>>> The original JIRA, WFLY-280[1], is to display the last 10 error messages only. To be honest I wouldn't find that very useful. To me if I'm looking for logs I want to see all logs, but that's not always so easy. Like the syslog-handler which doesn't log to a file so there is no way to read those messages back. >>>> >>>> The current plan for the last 10 error messages is we store messages in a queue that can be accessed via an operation. This works fine until the error message you're interested in is 11 or you want to see warning messages. >>>> >>>> Another option I had come up with is reading back the contents of the file, for example the server.log. This could be problematic too in that there is no way to filter information like only see error messages or only see warning messages. To solve this I have considered creating a JSON formatter so the results could be queried, but I don't think it should be a default which would mean it's not reliable for the console to assume it's getting back JSON. >>>> >>>> I've also thought about, haven't tested this and it may not work at all, creating a handler that uses websockets to send messages. I'm not sure how well this would work and it's possible it may not even work for bootstrap logging. >>>> >>>> With regards to audit logging, we're probably going to have to do something totally different from what we'll do in the logging subsystem since it doesn't use standard logging. >>>> >>>> I guess the bottom line is what does the console want to see? Do you want to see all raw text log messages? Do you want all messages but in a format like JSON that you can query/filter? Do you really want only the last 10 error messages only? All or none of these might be possible, but I really need to understand the needs before I can explore more in depth what the best option would be. >>>> >>>> [1]: https://issues.jboss.org/browse/WFLY-280 >>>> -- >>>> James R. Perkins >>>> Red Hat JBoss Middleware >>>> >>>> _______________________________________________ >>>> wildfly-dev mailing list >>>> wildfly-dev(a)lists.jboss.org >>>> https://lists.jboss.org/mailman/listinfo/wildfly-dev >>> -- >>> Jason T. Greene >>> WildFly Lead / JBoss EAP Platform Architect >>> JBoss, a division of Red Hat >>> >> -- >> James R. Perkins >> Red Hat JBoss Middleware >> > -- > Jason T. Greene > WildFly Lead / JBoss EAP Platform Architect > JBoss, a division of Red Hat > -- James R. Perkins Red Hat JBoss Middleware

That's my thinking too. The only complaint I've had about that solution is that it can't be filtered since it's just raw text. I had a working example operation that just took a file name and the number of bytes to read. To me this is the best solution, but it wouldn't have access to anything in a syslog or the console. Though that's okay IMO since they likely have other viewers for those. I'll resurrect that example. It shouldn't take all that long. One thing to note is it will only allow files to be read that are in the jboss.server.log.dir (and possibly only files that end in *.log?).

So in general I agree here, I think this is the best, the least invasive approach and the lowest performance hit. On 08/14/2013 10:36 AM, Jason Greene wrote: > IMO the best solution is a management operation that simply displays portions of the log file. The dependency on the log file is IMO not a problem because we support multiple appenders. > > On Aug 14, 2013, at 12:03 PM, James R. Perkins <jperkins(a)redhat.com&gt; wrote: > >> I had posted this to another list, but this is a more appropriate place for it. I think there needs to be a general discussion around this as it's been mentioned, at least to me, a few times here and there and I know Heiko raised the issue some time a go now. >> >> The original JIRA, WFLY-280[1], is to display the last 10 error messages only. To be honest I wouldn't find that very useful. To me if I'm looking for logs I want to see all logs, but that's not always so easy. Like the syslog-handler which doesn't log to a file so there is no way to read those messages back. >> >> The current plan for the last 10 error messages is we store messages in a queue that can be accessed via an operation. This works fine until the error message you're interested in is 11 or you want to see warning messages. >> >> Another option I had come up with is reading back the contents of the file, for example the server.log. This could be problematic too in that there is no way to filter information like only see error messages or only see warning messages. To solve this I have considered creating a JSON formatter so the results could be queried, but I don't think it should be a default which would mean it's not reliable for the console to assume it's getting back JSON. >> >> I've also thought about, haven't tested this and it may not work at all, creating a handler that uses websockets to send messages. I'm not sure how well this would work and it's possible it may not even work for bootstrap logging. >> >> With regards to audit logging, we're probably going to have to do something totally different from what we'll do in the logging subsystem since it doesn't use standard logging. >> >> I guess the bottom line is what does the console want to see? Do you want to see all raw text log messages? Do you want all messages but in a format like JSON that you can query/filter? Do you really want only the last 10 error messages only? All or none of these might be possible, but I really need to understand the needs before I can explore more in depth what the best option would be. >> >> [1]: https://issues.jboss.org/browse/WFLY-280 >> -- >> James R. Perkins >> Red Hat JBoss Middleware >> >> _______________________________________________ >> wildfly-dev mailing list >> wildfly-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/wildfly-dev > -- > Jason T. Greene > WildFly Lead / JBoss EAP Platform Architect > JBoss, a division of Red Hat >

On 08/14/2013 11:45 AM, Brian Stansberry wrote: > On 8/14/13 12:55 PM, James R. Perkins wrote: >> That's my thinking too. The only complaint I've had about that solution >> is that it can't be filtered since it's just raw text. I had a working >> example operation that just took a file name and the number of bytes to >> read. To me this is the best solution, but it wouldn't have access to >> anything in a syslog or the console. Though that's okay IMO since they >> likely have other viewers for those. >> >> I'll resurrect that example. It shouldn't take all that long. One thing >> to note is it will only allow files to be read that are in the >> jboss.server.log.dir (and possibly only files that end in *.log?). >> > That's only true if the viewer operation is not associated with a > management resource that maps to the log file (i.e. the appender > resource). > > I think it's a mistake to create any kind of viewer that can serve an > arbitrary file. > > For cases of serving previous versions of the file (i.e. ones before the > current with any kind of rolling appender) the API can expose a param > that allows the desired file to be chosen but only from a set of files > that the management resource knows it created. The only reason I thought of limiting it to the jboss.server.log.dir was there is no real way to know what handlers are file handlers. For example a custom-handler could be a file handler, but there is no real way to know that. Those files should be allowed to viewed, IMO, as well. On the other hand I do agree there is an issue with being able to read arbitrary files. A good example would be if the logs are placed in /var/log/ we don't want to allow access to any of the log files not created by WildFly. I guess what we could do, at least to start, is just add a read-log operation only to file handlers. Then we could just get the file name from the resource. Further down maybe we could add an attribute to the custom-handler to enable the operation.

> >> So in general I agree here, I think this is the best, the least >> invasive >> approach and the lowest performance hit. >> >> On 08/14/2013 10:36 AM, Jason Greene wrote: >>> IMO the best solution is a management operation that simply >>> displays portions of the log file. The dependency on the log file >>> is IMO not a problem because we support multiple appenders. >>> >>> On Aug 14, 2013, at 12:03 PM, James R. Perkins >>> <jperkins(a)redhat.com&gt; wrote: >>> >>>> I had posted this to another list, but this is a more appropriate >>>> place for it. I think there needs to be a general discussion >>>> around this as it's been mentioned, at least to me, a few times >>>> here and there and I know Heiko raised the issue some time a go now. >>>> >>>> The original JIRA, WFLY-280[1], is to display the last 10 error >>>> messages only. To be honest I wouldn't find that very useful. To >>>> me if I'm looking for logs I want to see all logs, but that's not >>>> always so easy. Like the syslog-handler which doesn't log to a >>>> file so there is no way to read those messages back. >>>> >>>> The current plan for the last 10 error messages is we store >>>> messages in a queue that can be accessed via an operation. This >>>> works fine until the error message you're interested in is 11 >>>> or you want to see warning messages. >>>> >>>> Another option I had come up with is reading back the contents of >>>> the file, for example the server.log. This could be problematic >>>> too in that there is no way to filter information like only see >>>> error messages or only see warning messages. To solve this I have >>>> considered creating a JSON formatter so the results could be >>>> queried, but I don't think it should be a default which would mean >>>> it's not reliable for the console to assume it's getting back JSON. >>>> >>>> I've also thought about, haven't tested this and it may not work >>>> at all, creating a handler that uses websockets to send messages. >>>> I'm not sure how well this would work and it's possible it may not >>>> even work for bootstrap logging. >>>> >>>> With regards to audit logging, we're probably going to have to do >>>> something totally different from what we'll do in the logging >>>> subsystem since it doesn't use standard logging. >>>> >>>> I guess the bottom line is what does the console want to see? Do >>>> you want to see all raw text log messages? Do you want all >>>> messages but in a format like JSON that you can query/filter? Do >>>> you really want only the last 10 error messages only? All or none >>>> of these might be possible, but I really need to understand the >>>> needs before I can explore more in depth what the best option >>>> would be. >>>> >>>> [1]: https://issues.jboss.org/browse/WFLY-280 >>>> -- >>>> James R. Perkins >>>> Red Hat JBoss Middleware >>>> >>>> _______________________________________________ >>>> wildfly-dev mailing list >>>> wildfly-dev(a)lists.jboss.org >>>> https://lists.jboss.org/mailman/listinfo/wildfly-dev >>> -- >>> Jason T. Greene >>> WildFly Lead / JBoss EAP Platform Architect >>> JBoss, a division of Red Hat >>> >

IMO the best solution is a management operation that simply displays portions of the log file. The dependency on the log file is IMO not a problem because we support multiple appenders.

On Aug 14, 2013, at 12:03 PM, James R. Perkins <jperkins(a)redhat.com&gt; wrote: > I had posted this to another list, but this is a more appropriate place for it. I think there needs to be a general discussion around this as it's been mentioned, at least to me, a few times here and there and I know Heiko raised the issue some time a go now. > > The original JIRA, WFLY-280[1], is to display the last 10 error messages only. To be honest I wouldn't find that very useful. To me if I'm looking for logs I want to see all logs, but that's not always so easy. Like the syslog-handler which doesn't log to a file so there is no way to read those messages back. > > The current plan for the last 10 error messages is we store messages in a queue that can be accessed via an operation. This works fine until the error message you're interested in is 11 or you want to see warning messages. > > Another option I had come up with is reading back the contents of the file, for example the server.log. This could be problematic too in that there is no way to filter information like only see error messages or only see warning messages. To solve this I have considered creating a JSON formatter so the results could be queried, but I don't think it should be a default which would mean it's not reliable for the console to assume it's getting back JSON. > > I've also thought about, haven't tested this and it may not work at all, creating a handler that uses websockets to send messages. I'm not sure how well this would work and it's possible it may not even work for bootstrap logging. > > With regards to audit logging, we're probably going to have to do something totally different from what we'll do in the logging subsystem since it doesn't use standard logging. > > I guess the bottom line is what does the console want to see? Do you want to see all raw text log messages? Do you want all messages but in a format like JSON that you can query/filter? Do you really want only the last 10 error messages only? All or none of these might be possible, but I really need to understand the needs before I can explore more in depth what the best option would be. > > [1]: https://issues.jboss.org/browse/WFLY-280 > -- > James R. Perkins > Red Hat JBoss Middleware > > _______________________________________________ > wildfly-dev mailing list > wildfly-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/wildfly-dev -- Jason T. Greene WildFly Lead / JBoss EAP Platform Architect JBoss, a division of Red Hat _______________________________________________ wildfly-dev mailing list wildfly-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/wildfly-dev

On 8/14/2013 1:36 PM, Jason Greene wrote: > IMO the best solution is a management operation that simply displays portions of the log file. The dependency on the log file is IMO not a problem because we support multiple appenders. I tend to agree. I dealt with this problem in the past and found that what I really needed was a just a copy (or partial copy) of the log file itself. So give me something that just lets me download the file. Then I can throw it into my favorite log file analyzer and deal with it that way. If we tried to embed our own analyzer into the console we'd never finish meeting all our customer's requirements.

What are the use cases for online reading of the server logs? If there are problems occurring on the application server (e.g. perhaps the cpu is pegged), reading logs online, could make the system even less responsive. If we just want to read the server logs as part of a health check, not requiring the server console to be working would be better. Should the reading of the logs instead be an external capability? Perhaps using the logs from the JBoss/WildFly Diagnostic Reporter output (archive) or some other archived copy of logs. Another compromise, add the WildFly Diagnostic Reporter (or at least the log collection part) to the management console (output archive is downloaded for local viewing). On 08/14/2013 01:03 PM, James R. Perkins wrote: > I had posted this to another list, but this is a more appropriate place > for it. I think there needs to be a general discussion around this as > it's been mentioned, at least to me, a few times here and there and I > know Heiko raised the issue some time a go now. > > The original JIRA, WFLY-280[1], is to display the last 10 error messages > only. To be honest I wouldn't find that very useful. To me if I'm > looking for logs I want to see all logs, but that's not always so easy. > Like the syslog-handler which doesn't log to a file so there is no way > to read those messages back. > > The current plan for the last 10 error messages is we store messages in > a queue that can be accessed via an operation. This works fine until the > error message you're interested in is 11 or you want to see warning > messages. > > Another option I had come up with is reading back the contents of the > file, for example the server.log. This could be problematic too in that > there is no way to filter information like only see error messages or > only see warning messages. To solve this I have considered creating a > JSON formatter so the results could be queried, but I don't think it > should be a default which would mean it's not reliable for the console > to assume it's getting back JSON. > > I've also thought about, haven't tested this and it may not work at all, > creating a handler that uses websockets to send messages. I'm not sure > how well this would work and it's possible it may not even work for > bootstrap logging. > > With regards to audit logging, we're probably going to have to do > something totally different from what we'll do in the logging subsystem > since it doesn't use standard logging. > > I guess the bottom line is what does the console want to see? Do you > want to see all raw text log messages? Do you want all messages but in a > format like JSON that you can query/filter? Do you really want only the > last 10 error messages only? All or none of these might be possible, but > I really need to understand the needs before I can explore more in depth > what the best option would be. > > [1]: https://issues.jboss.org/browse/WFLY-280 > > -- > James R. Perkins > Red Hat JBoss Middleware > > > > _______________________________________________ > wildfly-dev mailing list > wildfly-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/wildfly-dev > _______________________________________________ wildfly-dev mailing list wildfly-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/wildfly-dev

Just to add too in a domain it would be nice to have a central spot to view logs instead of having to ssh into various servers. On 08/14/2013 12:56 PM, Jason Greene wrote: > Mainly convenience. You deploy something it fails, you want to look at the log but don't feel like sshing into the server. As to performance the cost would be on request, and not more expensive then looking at the log file via ssh. > > On Aug 14, 2013, at 2:24 PM, Scott Marlow <smarlow(a)redhat.com&gt; wrote: > >> What are the use cases for online reading of the server logs? If there >> are problems occurring on the application server (e.g. perhaps the cpu >> is pegged), reading logs online, could make the system even less >> responsive. >> >> If we just want to read the server logs as part of a health check, not >> requiring the server console to be working would be better. >> >> Should the reading of the logs instead be an external capability? >> Perhaps using the logs from the JBoss/WildFly Diagnostic Reporter output >> (archive) or some other archived copy of logs. >> >> Another compromise, add the WildFly Diagnostic Reporter (or at least the >> log collection part) to the management console (output archive is >> downloaded for local viewing). >> >> On 08/14/2013 01:03 PM, James R. Perkins wrote: >>> I had posted this to another list, but this is a more appropriate place >>> for it. I think there needs to be a general discussion around this as >>> it's been mentioned, at least to me, a few times here and there and I >>> know Heiko raised the issue some time a go now. >>> >>> The original JIRA, WFLY-280[1], is to display the last 10 error messages >>> only. To be honest I wouldn't find that very useful. To me if I'm >>> looking for logs I want to see all logs, but that's not always so easy. >>> Like the syslog-handler which doesn't log to a file so there is no way >>> to read those messages back. >>> >>> The current plan for the last 10 error messages is we store messages in >>> a queue that can be accessed via an operation. This works fine until the >>> error message you're interested in is 11 or you want to see warning >>> messages. >>> >>> Another option I had come up with is reading back the contents of the >>> file, for example the server.log. This could be problematic too in that >>> there is no way to filter information like only see error messages or >>> only see warning messages. To solve this I have considered creating a >>> JSON formatter so the results could be queried, but I don't think it >>> should be a default which would mean it's not reliable for the console >>> to assume it's getting back JSON. >>> >>> I've also thought about, haven't tested this and it may not work at all, >>> creating a handler that uses websockets to send messages. I'm not sure >>> how well this would work and it's possible it may not even work for >>> bootstrap logging. >>> >>> With regards to audit logging, we're probably going to have to do >>> something totally different from what we'll do in the logging subsystem >>> since it doesn't use standard logging. >>> >>> I guess the bottom line is what does the console want to see? Do you >>> want to see all raw text log messages? Do you want all messages but in a >>> format like JSON that you can query/filter? Do you really want only the >>> last 10 error messages only? All or none of these might be possible, but >>> I really need to understand the needs before I can explore more in depth >>> what the best option would be. >>> >>> [1]: https://issues.jboss.org/browse/WFLY-280 >>> >>> -- >>> James R. Perkins >>> Red Hat JBoss Middleware >>> >>> >>> >>> _______________________________________________ >>> wildfly-dev mailing list >>> wildfly-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/wildfly-dev >>> >> _______________________________________________ >> wildfly-dev mailing list >> wildfly-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/wildfly-dev > -- > Jason T. Greene > WildFly Lead / JBoss EAP Platform Architect > JBoss, a division of Red Hat >

On 8/14/2013 5:20 PM, Brian Stansberry wrote: > Agreed. IMHO this is the most important driver for this feature. > > On 8/14/13 3:44 PM, James R. Perkins wrote: >> Just to add too in a domain it would be nice to have a central spot to >> view logs instead of having to ssh into various servers. Also realize that in many shops you don't have the option to ssh into various servers. If they are serious about security then access to the console is likely to be the only access you have. >> >> On 08/14/2013 12:56 PM, Jason Greene wrote: >>> Mainly convenience. You deploy something it fails, you want to look at the log but don't feel like sshing into the server. As to performance the cost would be on request, and not more expensive then looking at the log file via ssh. >>> >>> On Aug 14, 2013, at 2:24 PM, Scott Marlow <smarlow(a)redhat.com&gt; wrote: >>> >>>> What are the use cases for online reading of the server logs? If there >>>> are problems occurring on the application server (e.g. perhaps the cpu >>>> is pegged), reading logs online, could make the system even less >>>> responsive. >>>> >>>> If we just want to read the server logs as part of a health check, not >>>> requiring the server console to be working would be better. >>>> >>>> Should the reading of the logs instead be an external capability? >>>> Perhaps using the logs from the JBoss/WildFly Diagnostic Reporter output >>>> (archive) or some other archived copy of logs. >>>> >>>> Another compromise, add the WildFly Diagnostic Reporter (or at least the >>>> log collection part) to the management console (output archive is >>>> downloaded for local viewing). >>>> >>>> On 08/14/2013 01:03 PM, James R. Perkins wrote: >>>>> I had posted this to another list, but this is a more appropriate place >>>>> for it. I think there needs to be a general discussion around this as >>>>> it's been mentioned, at least to me, a few times here and there and I >>>>> know Heiko raised the issue some time a go now. >>>>> >>>>> The original JIRA, WFLY-280[1], is to display the last 10 error messages >>>>> only. To be honest I wouldn't find that very useful. To me if I'm >>>>> looking for logs I want to see all logs, but that's not always so easy. >>>>> Like the syslog-handler which doesn't log to a file so there is no way >>>>> to read those messages back. >>>>> >>>>> The current plan for the last 10 error messages is we store messages in >>>>> a queue that can be accessed via an operation. This works fine until the >>>>> error message you're interested in is 11 or you want to see warning >>>>> messages. >>>>> >>>>> Another option I had come up with is reading back the contents of the >>>>> file, for example the server.log. This could be problematic too in that >>>>> there is no way to filter information like only see error messages or >>>>> only see warning messages. To solve this I have considered creating a >>>>> JSON formatter so the results could be queried, but I don't think it >>>>> should be a default which would mean it's not reliable for the console >>>>> to assume it's getting back JSON. >>>>> >>>>> I've also thought about, haven't tested this and it may not work at all, >>>>> creating a handler that uses websockets to send messages. I'm not sure >>>>> how well this would work and it's possible it may not even work for >>>>> bootstrap logging. >>>>> >>>>> With regards to audit logging, we're probably going to have to do >>>>> something totally different from what we'll do in the logging subsystem >>>>> since it doesn't use standard logging. >>>>> >>>>> I guess the bottom line is what does the console want to see? Do you >>>>> want to see all raw text log messages? Do you want all messages but in a >>>>> format like JSON that you can query/filter? Do you really want only the >>>>> last 10 error messages only? All or none of these might be possible, but >>>>> I really need to understand the needs before I can explore more in depth >>>>> what the best option would be. >>>>> >>>>> [1]: https://issues.jboss.org/browse/WFLY-280 >>>>> >>>>> -- >>>>> James R. Perkins >>>>> Red Hat JBoss Middleware >>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> wildfly-dev mailing list >>>>> wildfly-dev(a)lists.jboss.org >>>>> https://lists.jboss.org/mailman/listinfo/wildfly-dev >>>>> >>>> _______________________________________________ >>>> wildfly-dev mailing list >>>> wildfly-dev(a)lists.jboss.org >>>> https://lists.jboss.org/mailman/listinfo/wildfly-dev >>> -- >>> Jason T. Greene >>> WildFly Lead / JBoss EAP Platform Architect >>> JBoss, a division of Red Hat >>> > _______________________________________________ wildfly-dev mailing list wildfly-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/wildfly-dev

On 08/15/2013 03:27 PM, Harald Pehl wrote: > I agree that we should focus on the most important use cases: > - Not able / not like to sshing into the server to look at the logs > - Deployment failed and you want to know why (staying in the console) > > As for the console I would suggest to start with a very limited set of features. No filtering, just show the last n log entries, where n is specified as part of the management operation. At first the user would see the last n log entries and can navigate further backwards in n-steps. For all deeper analysis the user should be able to download the full log. When showing the log to the user, I can think of some kind of syntax highlighting (see http://www.gnu.org/software/src-highlite/error.log.html for an example). The problem here is we don't know the last n entries if we read a file. We can read back the last n bytes or possibly lines, but that could contain portions of a stack trace. > > That said there are however use cases where parsing / filtering makes sense. The audit log has a well defined format (https://docs.jboss.org/author/display/WFLY8/Audit+logging#Auditlogging-JS...) and in the current implementation I'm using a master / detail view to display the audit log items. This might be a little easier for audit-logging since we have more control over what's happening. I have to look at the code, but we might just want to always write a formatted file in addition to other types like the syslog. > > .: Harald > > --- > Harald Pehl > JBoss by Red Hat > http://hpehl.info > > > > > Am 15.08.2013 um 03:11 schrieb ssilvert(a)redhat.com: > >> On 8/14/2013 5:20 PM, Brian Stansberry wrote: >>> Agreed. IMHO this is the most important driver for this feature. >>> >>> On 8/14/13 3:44 PM, James R. Perkins wrote: >>>> Just to add too in a domain it would be nice to have a central spot to >>>> view logs instead of having to ssh into various servers. >> Also realize that in many shops you don't have the option to ssh into >> various servers. If they are serious about security then access to the >> console is likely to be the only access you have. >> >>>> >>>> On 08/14/2013 12:56 PM, Jason Greene wrote: >>>>> Mainly convenience. You deploy something it fails, you want to look at the log but don't feel like sshing into the server. As to performance the cost would be on request, and not more expensive then looking at the log file via ssh. >>>>> >>>>> On Aug 14, 2013, at 2:24 PM, Scott Marlow <smarlow(a)redhat.com&gt; wrote: >>>>> >>>>>> What are the use cases for online reading of the server logs? If there >>>>>> are problems occurring on the application server (e.g. perhaps the cpu >>>>>> is pegged), reading logs online, could make the system even less >>>>>> responsive. >>>>>> >>>>>> If we just want to read the server logs as part of a health check, not >>>>>> requiring the server console to be working would be better. >>>>>> >>>>>> Should the reading of the logs instead be an external capability? >>>>>> Perhaps using the logs from the JBoss/WildFly Diagnostic Reporter output >>>>>> (archive) or some other archived copy of logs. >>>>>> >>>>>> Another compromise, add the WildFly Diagnostic Reporter (or at least the >>>>>> log collection part) to the management console (output archive is >>>>>> downloaded for local viewing). >>>>>> >>>>>> On 08/14/2013 01:03 PM, James R. Perkins wrote: >>>>>>> I had posted this to another list, but this is a more appropriate place >>>>>>> for it. I think there needs to be a general discussion around this as >>>>>>> it's been mentioned, at least to me, a few times here and there and I >>>>>>> know Heiko raised the issue some time a go now. >>>>>>> >>>>>>> The original JIRA, WFLY-280[1], is to display the last 10 error messages >>>>>>> only. To be honest I wouldn't find that very useful. To me if I'm >>>>>>> looking for logs I want to see all logs, but that's not always so easy. >>>>>>> Like the syslog-handler which doesn't log to a file so there is no way >>>>>>> to read those messages back. >>>>>>> >>>>>>> The current plan for the last 10 error messages is we store messages in >>>>>>> a queue that can be accessed via an operation. This works fine until the >>>>>>> error message you're interested in is 11 or you want to see warning >>>>>>> messages. >>>>>>> >>>>>>> Another option I had come up with is reading back the contents of the >>>>>>> file, for example the server.log. This could be problematic too in that >>>>>>> there is no way to filter information like only see error messages or >>>>>>> only see warning messages. To solve this I have considered creating a >>>>>>> JSON formatter so the results could be queried, but I don't think it >>>>>>> should be a default which would mean it's not reliable for the console >>>>>>> to assume it's getting back JSON. >>>>>>> >>>>>>> I've also thought about, haven't tested this and it may not work at all, >>>>>>> creating a handler that uses websockets to send messages. I'm not sure >>>>>>> how well this would work and it's possible it may not even work for >>>>>>> bootstrap logging. >>>>>>> >>>>>>> With regards to audit logging, we're probably going to have to do >>>>>>> something totally different from what we'll do in the logging subsystem >>>>>>> since it doesn't use standard logging. >>>>>>> >>>>>>> I guess the bottom line is what does the console want to see? Do you >>>>>>> want to see all raw text log messages? Do you want all messages but in a >>>>>>> format like JSON that you can query/filter? Do you really want only the >>>>>>> last 10 error messages only? All or none of these might be possible, but >>>>>>> I really need to understand the needs before I can explore more in depth >>>>>>> what the best option would be. >>>>>>> >>>>>>> [1]: https://issues.jboss.org/browse/WFLY-280 >>>>>>> >>>>>>> -- >>>>>>> James R. Perkins >>>>>>> Red Hat JBoss Middleware >>>>>>> >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> wildfly-dev mailing list >>>>>>> wildfly-dev(a)lists.jboss.org >>>>>>> https://lists.jboss.org/mailman/listinfo/wildfly-dev >>>>>> _______________________________________________ >>>>>> wildfly-dev mailing list >>>>>> wildfly-dev(a)lists.jboss.org >>>>>> https://lists.jboss.org/mailman/listinfo/wildfly-dev >>>>> -- >>>>> Jason T. Greene >>>>> WildFly Lead / JBoss EAP Platform Architect >>>>> JBoss, a division of Red Hat >> >> _______________________________________________ >> wildfly-dev mailing list >> wildfly-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/wildfly-dev > > > > _______________________________________________ > wildfly-dev mailing list > wildfly-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/wildfly-dev -- James R. Perkins Red Hat JBoss Middleware _______________________________________________ wildfly-dev mailing list wildfly-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/wildfly-dev

I had posted this to another list, but this is a more appropriate place for it. I think there needs to be a general discussion around this as it's been mentioned, at least to me, a few times here and there and I know Heiko raised the issue some time a go now. The original JIRA, WFLY-280[1], is to display the last 10 error messages only. To be honest I wouldn't find that very useful. To me if I'm looking for logs I want to see all logs, but that's not always so easy. Like the syslog-handler which doesn't log to a file so there is no way to read those messages back. The current plan for the last 10 error messages is we store messages in a queue that can be accessed via an operation. This works fine until the error message you're interested in is 11 or you want to see warning messages. Another option I had come up with is reading back the contents of the file, for example the server.log. This could be problematic too in that there is no way to filter information like only see error messages or only see warning messages. To solve this I have considered creating a JSON formatter so the results could be queried, but I don't think it should be a default which would mean it's not reliable for the console to assume it's getting back JSON. I've also thought about, haven't tested this and it may not work at all, creating a handler that uses websockets to send messages. I'm not sure how well this would work and it's possible it may not even work for bootstrap logging. With regards to audit logging, we're probably going to have to do something totally different from what we'll do in the logging subsystem since it doesn't use standard logging. I guess the bottom line is what does the console want to see? Do you want to see all raw text log messages? Do you want all messages but in a format like JSON that you can query/filter? Do you really want only the last 10 error messages only? All or none of these might be possible, but I really need to understand the needs before I can explore more in depth what the best option would be. [1]: https://issues.jboss.org/browse/WFLY-280 -- James R. Perkins Red Hat JBoss Middleware

1) Could it have a "read-log-file()" without name= specified, which would read the "current" log file?

2) Regarding security - what, besides logs, do we expect to be in the log dir? Could the admin block it by setting write-only rights? Ondra On 25.9.2013 02:40, James R. Perkins wrote: > I'm replying to this old thread to reopen this conversation about > reading log files. I've complete some work [1] on reading log files > via an operation. This is not exactly like the JIRA suggests where it > would only read the last 10 error messages. All this change allows is > the raw contents of the file to be read. The idea is this could be > used to read the entire contents of the log file as a whole, or in > chunks. > > What I've done is added two new operations list-log-files and > read-log-file. > > The list-log-files simply lists all files in the > jboss.server.log.dir. This may or may not be a good idea really. I > can see some potential security risks here mainly just seeing files > that may contain sensitive data. One way I've thought of to get > around that is read the logging subsystem model and only show files > from known types like the file-handlers. The main issue with that is > there is no good way to get this to work for custom-handlers. > > The read-log-file simple does what it says and reads the contents of > a log file line by line. Reading line by line should work for the > most part unless the an non-standard line delimiter is used. There > are 5 options for this option; > > * name (required): the name of the log file to read > * encoding: the encoding for the log file > * lines: the number of lines to read, defaults to 10 > * skip: the number of lines to skip before adding the results > * tail: true to read from the bottom up, default is true > > The result of this is just a list of lines with the \n or \r\n > stripped. Just to clarify too a line means a line in the file, not a > log record e.g. stack traces are generally composed of multiple lines. > > So this begs the question, will this work for what we want? What > concerns does anyone else have? > > I have not yet submitted a PR yet as I wanted to get some feedback > before we bake it in. > > > [1]: https://github.com/jamezp/wildfly/compare/WFLY-280-read > > > On 08/14/2013 10:03 AM, James R. Perkins wrote: >> I had posted this to another list, but this is a more appropriate >> place for it. I think there needs to be a general discussion around >> this as it's been mentioned, at least to me, a few times here and >> there and I know Heiko raised the issue some time a go now. >> >> The original JIRA, WFLY-280[1], is to display the last 10 error >> messages only. To be honest I wouldn't find that very useful. To me >> if I'm looking for logs I want to see all logs, but that's not >> always so easy. Like the syslog-handler which doesn't log to a file >> so there is no way to read those messages back. >> >> The current plan for the last 10 error messages is we store messages >> in a queue that can be accessed via an operation. This works fine >> until the error message you're interested in is 11 or you want to >> see warning messages. >> >> Another option I had come up with is reading back the contents of >> the file, for example the server.log. This could be problematic too >> in that there is no way to filter information like only see error >> messages or only see warning messages. To solve this I have >> considered creating a JSON formatter so the results could be >> queried, but I don't think it should be a default which would mean >> it's not reliable for the console to assume it's getting back JSON. >> >> I've also thought about, haven't tested this and it may not work at >> all, creating a handler that uses websockets to send messages. I'm >> not sure how well this would work and it's possible it may not even >> work for bootstrap logging. >> >> With regards to audit logging, we're probably going to have to do >> something totally different from what we'll do in the logging >> subsystem since it doesn't use standard logging. >> >> I guess the bottom line is what does the console want to see? Do you >> want to see all raw text log messages? Do you want all messages but >> in a format like JSON that you can query/filter? Do you really want >> only the last 10 error messages only? All or none of these might be >> possible, but I really need to understand the needs before I can >> explore more in depth what the best option would be. >> >> [1]: https://issues.jboss.org/browse/WFLY-280 >> -- >> James R. Perkins >> Red Hat JBoss Middleware > > -- > James R. Perkins > Red Hat JBoss Middleware > > > _______________________________________________ > wildfly-dev mailing list > wildfly-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/wildfly-dev _______________________________________________ wildfly-dev mailing list wildfly-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/wildfly-dev

Current log could be the server.log in a standard setup. If someone has done something more advanced in their logging setup, the name becomes necessary On 25 Sep 2013, at 08:34, Jaikiran Pai wrote: > On Wednesday 25 September 2013 09:25 AM, Ondrej Zizka wrote: >> 1) Could it have a "read-log-file()" without name= specified, which would read the "current" log file? > Given the way logger categories and appenders/handlers interact, within a logging framework, I don't think there's any notion of "current log file". It's a very valid scenario where a single logging category can be backed by different appenders (some of them file appenders) with different attributes and each such appender writing out to a different file. So having a name of the log file you want to view, becomes necessary. > > -Jaikiran > >> 2) Regarding security - what, besides logs, do we expect to be in the log dir? Could the admin block it by setting write-only rights? >> >> Ondra >> >> On 25.9.2013 02:40, James R. Perkins wrote: >>> I'm replying to this old thread to reopen this conversation about reading log files. I've complete some work [1] on reading log files via an operation. This is not exactly like the JIRA suggests where it would only read the last 10 error messages. All this change allows is the raw contents of the file to be read. The idea is this could be used to read the entire contents of the log file as a whole, or in chunks. >>> >>> What I've done is added two new operations list-log-files and read-log-file. >>> >>> The list-log-files simply lists all files in the jboss.server.log.dir. This may or may not be a good idea really. I can see some potential security risks here mainly just seeing files that may contain sensitive data. One way I've thought of to get around that is read the logging subsystem model and only show files from known types like the file-handlers. The main issue with that is there is no good way to get this to work for custom-handlers. >>> >>> The read-log-file simple does what it says and reads the contents of a log file line by line. Reading line by line should work for the most part unless the an non-standard line delimiter is used. There are 5 options for this option; >>> • name (required): the name of the log file to read >>> • encoding: the encoding for the log file >>> • lines: the number of lines to read, defaults to 10 >>> • skip: the number of lines to skip before adding the results >>> • tail: true to read from the bottom up, default is true >>> The result of this is just a list of lines with the \n or \r\n stripped. Just to clarify too a line means a line in the file, not a log record e.g. stack traces are generally composed of multiple lines. >>> >>> So this begs the question, will this work for what we want? What concerns does anyone else have? >>> >>> I have not yet submitted a PR yet as I wanted to get some feedback before we bake it in. >>> >>> >>> [1]: https://github.com/jamezp/wildfly/compare/WFLY-280-read >>> >>> >>> On 08/14/2013 10:03 AM, James R. Perkins wrote: >>>> I had posted this to another list, but this is a more appropriate place for it. I think there needs to be a general discussion around this as it's been mentioned, at least to me, a few times here and there and I know Heiko raised the issue some time a go now. >>>> >>>> The original JIRA, WFLY-280[1], is to display the last 10 error messages only. To be honest I wouldn't find that very useful. To me if I'm looking for logs I want to see all logs, but that's not always so easy. Like the syslog-handler which doesn't log to a file so there is no way to read those messages back. >>>> >>>> The current plan for the last 10 error messages is we store messages in a queue that can be accessed via an operation. This works fine until the error message you're interested in is 11 or you want to see warning messages. >>>> >>>> Another option I had come up with is reading back the contents of the file, for example the server.log. This could be problematic too in that there is no way to filter information like only see error messages or only see warning messages. To solve this I have considered creating a JSON formatter so the results could be queried, but I don't think it should be a default which would mean it's not reliable for the console to assume it's getting back JSON. >>>> >>>> I've also thought about, haven't tested this and it may not work at all, creating a handler that uses websockets to send messages. I'm not sure how well this would work and it's possible it may not even work for bootstrap logging. >>>> >>>> With regards to audit logging, we're probably going to have to do something totally different from what we'll do in the logging subsystem since it doesn't use standard logging. >>>> >>>> I guess the bottom line is what does the console want to see? Do you want to see all raw text log messages? Do you want all messages but in a format like JSON that you can query/filter? Do you really want only the last 10 error messages only? All or none of these might be possible, but I really need to understand the needs before I can explore more in depth what the best option would be. >>>> >>>> [1]: https://issues.jboss.org/browse/WFLY-280 >>>> -- >>>> James R. Perkins >>>> Red Hat JBoss Middleware >>>> >>> >>> -- >>> James R. Perkins >>> Red Hat JBoss Middleware >>> >>> >>> >>> >>> _______________________________________________ >>> wildfly-dev mailing list >>> >>> wildfly-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/wildfly-dev >> >> >> >> _______________________________________________ >> wildfly-dev mailing list >> >> wildfly-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/wildfly-dev > _______________________________________________ > wildfly-dev mailing list > wildfly-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/wildfly-dev

Just a thought regarding your PR https://github.com/wildfly/wildfly/pull/5156 The thing that stands out to me is that you're able to read the first 100 lines, e.g. :read-log-file(name=server.log, lines=100, tail=false) or the last 100 lines :read-log-file(name=server.log, lines=100, tail=true) Would it also be beneficial to also be allowed to start somewhere in the middle, e.g :read-log-file(name=server.log, lines=100, start=10235) or (19:38:01 today)

:read-log-file(name=server.log, lines=100, start-time=20131008-10:38:01) I'm not saying it MUST be like that but it might be worth discussing On 25 Sep 2013, at 16:36, Ondrej Zizka wrote: > Right, by "current" I meant to read the logging setup and if there's > just one file appender, take it as default. Or, if that would be > inappropriate mixing of abstraction layers, just "server.log". > > > On 25.9.2013 12:06, Kabir Khan wrote: >> Current log could be the server.log in a standard setup. If someone has done something more advanced in their logging setup, the name becomes necessary >> On 25 Sep 2013, at 08:34, Jaikiran Pai wrote: >> >>> On Wednesday 25 September 2013 09:25 AM, Ondrej Zizka wrote: >>>> 1) Could it have a "read-log-file()" without name= specified, which would read the "current" log file? >>> Given the way logger categories and appenders/handlers interact, within a logging framework, I don't think there's any notion of "current log file". It's a very valid scenario where a single logging category can be backed by different appenders (some of them file appenders) with different attributes and each such appender writing out to a different file. So having a name of the log file you want to view, becomes necessary. >>> >>> -Jaikiran >>> >>>> 2) Regarding security - what, besides logs, do we expect to be in the log dir? Could the admin block it by setting write-only rights? >>>> >>>> Ondra >>>> >>>> On 25.9.2013 02:40, James R. Perkins wrote: >>>>> I'm replying to this old thread to reopen this conversation about reading log files. I've complete some work [1] on reading log files via an operation. This is not exactly like the JIRA suggests where it would only read the last 10 error messages. All this change allows is the raw contents of the file to be read. The idea is this could be used to read the entire contents of the log file as a whole, or in chunks. >>>>> >>>>> What I've done is added two new operations list-log-files and read-log-file. >>>>> >>>>> The list-log-files simply lists all files in the jboss.server.log.dir. This may or may not be a good idea really. I can see some potential security risks here mainly just seeing files that may contain sensitive data. One way I've thought of to get around that is read the logging subsystem model and only show files from known types like the file-handlers. The main issue with that is there is no good way to get this to work for custom-handlers. >>>>> >>>>> The read-log-file simple does what it says and reads the contents of a log file line by line. Reading line by line should work for the most part unless the an non-standard line delimiter is used. There are 5 options for this option; >>>>> • name (required): the name of the log file to read >>>>> • encoding: the encoding for the log file >>>>> • lines: the number of lines to read, defaults to 10 >>>>> • skip: the number of lines to skip before adding the results >>>>> • tail: true to read from the bottom up, default is true >>>>> The result of this is just a list of lines with the \n or \r\n stripped. Just to clarify too a line means a line in the file, not a log record e.g. stack traces are generally composed of multiple lines. >>>>> >>>>> So this begs the question, will this work for what we want? What concerns does anyone else have? >>>>> >>>>> I have not yet submitted a PR yet as I wanted to get some feedback before we bake it in. >>>>> >>>>> >>>>> [1]: https://github.com/jamezp/wildfly/compare/WFLY-280-read >>>>> >>>>> >>>>> On 08/14/2013 10:03 AM, James R. Perkins wrote: >>>>>> I had posted this to another list, but this is a more appropriate place for it. I think there needs to be a general discussion around this as it's been mentioned, at least to me, a few times here and there and I know Heiko raised the issue some time a go now. >>>>>> >>>>>> The original JIRA, WFLY-280[1], is to display the last 10 error messages only. To be honest I wouldn't find that very useful. To me if I'm looking for logs I want to see all logs, but that's not always so easy. Like the syslog-handler which doesn't log to a file so there is no way to read those messages back. >>>>>> >>>>>> The current plan for the last 10 error messages is we store messages in a queue that can be accessed via an operation. This works fine until the error message you're interested in is 11 or you want to see warning messages. >>>>>> >>>>>> Another option I had come up with is reading back the contents of the file, for example the server.log. This could be problematic too in that there is no way to filter information like only see error messages or only see warning messages. To solve this I have considered creating a JSON formatter so the results could be queried, but I don't think it should be a default which would mean it's not reliable for the console to assume it's getting back JSON. >>>>>> >>>>>> I've also thought about, haven't tested this and it may not work at all, creating a handler that uses websockets to send messages. I'm not sure how well this would work and it's possible it may not even work for bootstrap logging. >>>>>> >>>>>> With regards to audit logging, we're probably going to have to do something totally different from what we'll do in the logging subsystem since it doesn't use standard logging. >>>>>> >>>>>> I guess the bottom line is what does the console want to see? Do you want to see all raw text log messages? Do you want all messages but in a format like JSON that you can query/filter? Do you really want only the last 10 error messages only? All or none of these might be possible, but I really need to understand the needs before I can explore more in depth what the best option would be. >>>>>> >>>>>> [1]: https://issues.jboss.org/browse/WFLY-280 >>>>>> -- >>>>>> James R. Perkins >>>>>> Red Hat JBoss Middleware >>>>>> >>>>> >>>>> -- >>>>> James R. Perkins >>>>> Red Hat JBoss Middleware >>>>> >>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> wildfly-dev mailing list >>>>> >>>>> wildfly-dev(a)lists.jboss.org >>>>> https://lists.jboss.org/mailman/listinfo/wildfly-dev >>>> >>>> >>>> >>>> _______________________________________________ >>>> wildfly-dev mailing list >>>> >>>> wildfly-dev(a)lists.jboss.org >>>> https://lists.jboss.org/mailman/listinfo/wildfly-dev >>> _______________________________________________ >>> wildfly-dev mailing list >>> wildfly-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/wildfly-dev > > _______________________________________________ > wildfly-dev mailing list > wildfly-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/wildfly-dev

On 10/08/2013 03:40 AM, Kabir Khan wrote: > Just a thought regarding your PR https://github.com/wildfly/wildfly/pull/5156 > > The thing that stands out to me is that you're able to read the first 100 lines, e.g. > :read-log-file(name=server.log, lines=100, tail=false) > or the last 100 lines > :read-log-file(name=server.log, lines=100, tail=true) > > Would it also be beneficial to also be allowed to start somewhere in the middle, e.g > :read-log-file(name=server.log, lines=100, start=10235) There is an option skip that will allows something similar. The following would read read the last 100 lines _after_ 50 lines are skipped from the bottom. So in a 200 line this would read lines 50 to line 150. :read-log-file(name=server.log, lines=100, skip=50, tail=true) The following would read from line 50 to line 150 :read-log-file(name=server.log, lines=100, skip=50, tail=false) I am using a BufferedReader so if an odd line terminator is used it will likely not work so well. I considered allowing for the line terminator to be defined in the operation, but it would require analyzing bytes for the terminator and seemed like more work than it was worth. I would guess the majority of users use the standard /n or /r/n terminators. > or (19:38:01 today) > :read-log-file(name=server.log, lines=100, start-time=20131008-10:38:01) The date would be rather difficult. We're not parsing the log strings just reading the raw text and sending it back. > > I'm not saying it MUST be like that but it might be worth discussing > > On 25 Sep 2013, at 16:36, Ondrej Zizka wrote: > >> Right, by "current" I meant to read the logging setup and if there's >> just one file appender, take it as default. Or, if that would be >> inappropriate mixing of abstraction layers, just "server.log". >> >> >> On 25.9.2013 12:06, Kabir Khan wrote: >>> Current log could be the server.log in a standard setup. If someone has done something more advanced in their logging setup, the name becomes necessary >>> On 25 Sep 2013, at 08:34, Jaikiran Pai wrote: >>> >>>> On Wednesday 25 September 2013 09:25 AM, Ondrej Zizka wrote: >>>>> 1) Could it have a "read-log-file()" without name= specified, which would read the "current" log file? >>>> Given the way logger categories and appenders/handlers interact, within a logging framework, I don't think there's any notion of "current log file". It's a very valid scenario where a single logging category can be backed by different appenders (some of them file appenders) with different attributes and each such appender writing out to a different file. So having a name of the log file you want to view, becomes necessary. >>>> >>>> -Jaikiran >>>> >>>>> 2) Regarding security - what, besides logs, do we expect to be in the log dir? Could the admin block it by setting write-only rights? >>>>> >>>>> Ondra >>>>> >>>>> On 25.9.2013 02:40, James R. Perkins wrote: >>>>>> I'm replying to this old thread to reopen this conversation about reading log files. I've complete some work [1] on reading log files via an operation. This is not exactly like the JIRA suggests where it would only read the last 10 error messages. All this change allows is the raw contents of the file to be read. The idea is this could be used to read the entire contents of the log file as a whole, or in chunks. >>>>>> >>>>>> What I've done is added two new operations list-log-files and read-log-file. >>>>>> >>>>>> The list-log-files simply lists all files in the jboss.server.log.dir. This may or may not be a good idea really. I can see some potential security risks here mainly just seeing files that may contain sensitive data. One way I've thought of to get around that is read the logging subsystem model and only show files from known types like the file-handlers. The main issue with that is there is no good way to get this to work for custom-handlers. >>>>>> >>>>>> The read-log-file simple does what it says and reads the contents of a log file line by line. Reading line by line should work for the most part unless the an non-standard line delimiter is used. There are 5 options for this option; >>>>>> • name (required): the name of the log file to read >>>>>> • encoding: the encoding for the log file >>>>>> • lines: the number of lines to read, defaults to 10 >>>>>> • skip: the number of lines to skip before adding the results >>>>>> • tail: true to read from the bottom up, default is true >>>>>> The result of this is just a list of lines with the \n or \r\n stripped. Just to clarify too a line means a line in the file, not a log record e.g. stack traces are generally composed of multiple lines. >>>>>> >>>>>> So this begs the question, will this work for what we want? What concerns does anyone else have? >>>>>> >>>>>> I have not yet submitted a PR yet as I wanted to get some feedback before we bake it in. >>>>>> >>>>>> >>>>>> [1]: https://github.com/jamezp/wildfly/compare/WFLY-280-read >>>>>> >>>>>> >>>>>> On 08/14/2013 10:03 AM, James R. Perkins wrote: >>>>>>> I had posted this to another list, but this is a more appropriate place for it. I think there needs to be a general discussion around this as it's been mentioned, at least to me, a few times here and there and I know Heiko raised the issue some time a go now. >>>>>>> >>>>>>> The original JIRA, WFLY-280[1], is to display the last 10 error messages only. To be honest I wouldn't find that very useful. To me if I'm looking for logs I want to see all logs, but that's not always so easy. Like the syslog-handler which doesn't log to a file so there is no way to read those messages back. >>>>>>> >>>>>>> The current plan for the last 10 error messages is we store messages in a queue that can be accessed via an operation. This works fine until the error message you're interested in is 11 or you want to see warning messages. >>>>>>> >>>>>>> Another option I had come up with is reading back the contents of the file, for example the server.log. This could be problematic too in that there is no way to filter information like only see error messages or only see warning messages. To solve this I have considered creating a JSON formatter so the results could be queried, but I don't think it should be a default which would mean it's not reliable for the console to assume it's getting back JSON. >>>>>>> >>>>>>> I've also thought about, haven't tested this and it may not work at all, creating a handler that uses websockets to send messages. I'm not sure how well this would work and it's possible it may not even work for bootstrap logging. >>>>>>> >>>>>>> With regards to audit logging, we're probably going to have to do something totally different from what we'll do in the logging subsystem since it doesn't use standard logging. >>>>>>> >>>>>>> I guess the bottom line is what does the console want to see? Do you want to see all raw text log messages? Do you want all messages but in a format like JSON that you can query/filter? Do you really want only the last 10 error messages only? All or none of these might be possible, but I really need to understand the needs before I can explore more in depth what the best option would be. >>>>>>> >>>>>>> [1]: https://issues.jboss.org/browse/WFLY-280 >>>>>>> -- >>>>>>> James R. Perkins >>>>>>> Red Hat JBoss Middleware >>>>>>> >>>>>> -- >>>>>> James R. Perkins >>>>>> Red Hat JBoss Middleware >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> wildfly-dev mailing list >>>>>> >>>>>> wildfly-dev(a)lists.jboss.org >>>>>> https://lists.jboss.org/mailman/listinfo/wildfly-dev >>>>> >>>>> >>>>> _______________________________________________ >>>>> wildfly-dev mailing list >>>>> >>>>> wildfly-dev(a)lists.jboss.org >>>>> https://lists.jboss.org/mailman/listinfo/wildfly-dev >>>> _______________________________________________ >>>> wildfly-dev mailing list >>>> wildfly-dev(a)lists.jboss.org >>>> https://lists.jboss.org/mailman/listinfo/wildfly-dev >> _______________________________________________ >> wildfly-dev mailing list >> wildfly-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/wildfly-dev

I've said it before and I'll say it again. Using the format to "intelligently" parse the logs looks like a good idea. It is actually a bad idea. A bad idea that looks like a good idea, or maybe even a "good enough" idea. Maybe it looks like an "easy" idea. But it's bad. Bad, and also not good. It will result in bug reports that nobody will bother fixing or implementing because they will be reasonable yet unfeasible to fix. Here are some implementation options that I *won't* immediately kill: 1) Treat the files in the log directory as opaque things that the server does not pretend to have interior knowledge of - in other words, provide operations that work on the file as a whole (or by things which make sense at a file level, like line number or dumb string grep) which will work 100% of the time 2) Provide in-memory (object) or on-disk (structured binary) storage of log record objects with intelligent search capabilities which will work 100% of the time Here are some implementation options that I *will* immediately kill: 1) Try to use the format to figure out record boundaries 2) Try to use the format to figure out the date of the record 3) Try to use the format to extract exception information 4) Any and all other format-based ideas not mentioned here The common theme is - it has to work 100% of the time. No excuses. On 10/08/2013 10:55 AM, James R. Perkins wrote: > On 10/08/2013 03:40 AM, Kabir Khan wrote: >> Just a thought regarding your PR https://github.com/wildfly/wildfly/pull/5156 >> >> The thing that stands out to me is that you're able to read the first 100 lines, e.g. >> :read-log-file(name=server.log, lines=100, tail=false) >> or the last 100 lines >> :read-log-file(name=server.log, lines=100, tail=true) >> >> Would it also be beneficial to also be allowed to start somewhere in the middle, e.g >> :read-log-file(name=server.log, lines=100, start=10235) > There is an option skip that will allows something similar. The > following would read read the last 100 lines _after_ 50 lines are > skipped from the bottom. So in a 200 line this would read lines 50 to > line 150. > :read-log-file(name=server.log, lines=100, skip=50, tail=true) > > The following would read from line 50 to line 150 > :read-log-file(name=server.log, lines=100, skip=50, tail=false) > > I am using a BufferedReader so if an odd line terminator is used it will > likely not work so well. I considered allowing for the line terminator > to be defined in the operation, but it would require analyzing bytes for > the terminator and seemed like more work than it was worth. I would > guess the majority of users use the standard /n or /r/n terminators. >> or (19:38:01 today) >> :read-log-file(name=server.log, lines=100, start-time=20131008-10:38:01) > The date would be rather difficult. We're not parsing the log strings > just reading the raw text and sending it back. >> I'm not saying it MUST be like that but it might be worth discussing >> >> On 25 Sep 2013, at 16:36, Ondrej Zizka wrote: >> >>> Right, by "current" I meant to read the logging setup and if there's >>> just one file appender, take it as default. Or, if that would be >>> inappropriate mixing of abstraction layers, just "server.log". >>> >>> >>> On 25.9.2013 12:06, Kabir Khan wrote: >>>> Current log could be the server.log in a standard setup. If someone has done something more advanced in their logging setup, the name becomes necessary >>>> On 25 Sep 2013, at 08:34, Jaikiran Pai wrote: >>>> >>>>> On Wednesday 25 September 2013 09:25 AM, Ondrej Zizka wrote: >>>>>> 1) Could it have a "read-log-file()" without name= specified, which would read the "current" log file? >>>>> Given the way logger categories and appenders/handlers interact, within a logging framework, I don't think there's any notion of "current log file". It's a very valid scenario where a single logging category can be backed by different appenders (some of them file appenders) with different attributes and each such appender writing out to a different file. So having a name of the log file you want to view, becomes necessary. >>>>> >>>>> -Jaikiran >>>>> >>>>>> 2) Regarding security - what, besides logs, do we expect to be in the log dir? Could the admin block it by setting write-only rights? >>>>>> >>>>>> Ondra >>>>>> >>>>>> On 25.9.2013 02:40, James R. Perkins wrote: >>>>>>> I'm replying to this old thread to reopen this conversation about reading log files. I've complete some work [1] on reading log files via an operation. This is not exactly like the JIRA suggests where it would only read the last 10 error messages. All this change allows is the raw contents of the file to be read. The idea is this could be used to read the entire contents of the log file as a whole, or in chunks. >>>>>>> >>>>>>> What I've done is added two new operations list-log-files and read-log-file. >>>>>>> >>>>>>> The list-log-files simply lists all files in the jboss.server.log.dir. This may or may not be a good idea really. I can see some potential security risks here mainly just seeing files that may contain sensitive data. One way I've thought of to get around that is read the logging subsystem model and only show files from known types like the file-handlers. The main issue with that is there is no good way to get this to work for custom-handlers. >>>>>>> >>>>>>> The read-log-file simple does what it says and reads the contents of a log file line by line. Reading line by line should work for the most part unless the an non-standard line delimiter is used. There are 5 options for this option; >>>>>>> • name (required): the name of the log file to read >>>>>>> • encoding: the encoding for the log file >>>>>>> • lines: the number of lines to read, defaults to 10 >>>>>>> • skip: the number of lines to skip before adding the results >>>>>>> • tail: true to read from the bottom up, default is true >>>>>>> The result of this is just a list of lines with the \n or \r\n stripped. Just to clarify too a line means a line in the file, not a log record e.g. stack traces are generally composed of multiple lines. >>>>>>> >>>>>>> So this begs the question, will this work for what we want? What concerns does anyone else have? >>>>>>> >>>>>>> I have not yet submitted a PR yet as I wanted to get some feedback before we bake it in. >>>>>>> >>>>>>> >>>>>>> [1]: https://github.com/jamezp/wildfly/compare/WFLY-280-read >>>>>>> >>>>>>> >>>>>>> On 08/14/2013 10:03 AM, James R. Perkins wrote: >>>>>>>> I had posted this to another list, but this is a more appropriate place for it. I think there needs to be a general discussion around this as it's been mentioned, at least to me, a few times here and there and I know Heiko raised the issue some time a go now. >>>>>>>> >>>>>>>> The original JIRA, WFLY-280[1], is to display the last 10 error messages only. To be honest I wouldn't find that very useful. To me if I'm looking for logs I want to see all logs, but that's not always so easy. Like the syslog-handler which doesn't log to a file so there is no way to read those messages back. >>>>>>>> >>>>>>>> The current plan for the last 10 error messages is we store messages in a queue that can be accessed via an operation. This works fine until the error message you're interested in is 11 or you want to see warning messages. >>>>>>>> >>>>>>>> Another option I had come up with is reading back the contents of the file, for example the server.log. This could be problematic too in that there is no way to filter information like only see error messages or only see warning messages. To solve this I have considered creating a JSON formatter so the results could be queried, but I don't think it should be a default which would mean it's not reliable for the console to assume it's getting back JSON. >>>>>>>> >>>>>>>> I've also thought about, haven't tested this and it may not work at all, creating a handler that uses websockets to send messages. I'm not sure how well this would work and it's possible it may not even work for bootstrap logging. >>>>>>>> >>>>>>>> With regards to audit logging, we're probably going to have to do something totally different from what we'll do in the logging subsystem since it doesn't use standard logging. >>>>>>>> >>>>>>>> I guess the bottom line is what does the console want to see? Do you want to see all raw text log messages? Do you want all messages but in a format like JSON that you can query/filter? Do you really want only the last 10 error messages only? All or none of these might be possible, but I really need to understand the needs before I can explore more in depth what the best option would be. >>>>>>>> >>>>>>>> [1]: https://issues.jboss.org/browse/WFLY-280 >>>>>>>> -- >>>>>>>> James R. Perkins >>>>>>>> Red Hat JBoss Middleware >>>>>>>> >>>>>>> -- >>>>>>> James R. Perkins >>>>>>> Red Hat JBoss Middleware >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> wildfly-dev mailing list >>>>>>> >>>>>>> wildfly-dev(a)lists.jboss.org >>>>>>> https://lists.jboss.org/mailman/listinfo/wildfly-dev >>>>>> >>>>>> _______________________________________________ >>>>>> wildfly-dev mailing list >>>>>> >>>>>> wildfly-dev(a)lists.jboss.org >>>>>> https://lists.jboss.org/mailman/listinfo/wildfly-dev >>>>> _______________________________________________ >>>>> wildfly-dev mailing list >>>>> wildfly-dev(a)lists.jboss.org >>>>> https://lists.jboss.org/mailman/listinfo/wildfly-dev >>> _______________________________________________ >>> wildfly-dev mailing list >>> wildfly-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/wildfly-dev

How about storing the offsets of the records. Then the operation would read this "index" and instead of format-based parsing, it would simply cut the right part. Simple, fast in both write and read, and doesn't consume too much memory/storage. Ondra On 8.10.2013 18:57, James R. Perkins wrote: > I'm definitely not trying to parse any of the log message format. I am > using a BufferedReader.readLine() which I was a little iffy on, but went > with "lines" as opposed to reading x bytes. Obviously from a management > standpoint reading the number of bytes makes more sense as we're not > relying on any kind of normal line terminator being used. From a user > standpoint though I think it makes less sense and the number of lines > makes more sense. > > I suppose we could have two operations so the web console could use the > one that reads bytes and then have one a simple tail like one that reads > by line. > > On 10/08/2013 09:26 AM, David M. Lloyd wrote: >> I've said it before and I'll say it again. Using the format to >> "intelligently" parse the logs looks like a good idea. It is >> actually a >> bad idea. A bad idea that looks like a good idea, or maybe even a >> "good >> enough" idea. Maybe it looks like an "easy" idea. But it's bad. Bad, >> and also not good. It will result in bug reports that nobody will >> bother fixing or implementing because they will be reasonable yet >> unfeasible to fix. >> >> Here are some implementation options that I *won't* immediately kill: >> >> 1) Treat the files in the log directory as opaque things that the >> server >> does not pretend to have interior knowledge of - in other words, >> provide >> operations that work on the file as a whole (or by things which make >> sense at a file level, like line number or dumb string grep) which will >> work 100% of the time >> 2) Provide in-memory (object) or on-disk (structured binary) storage of >> log record objects with intelligent search capabilities which will work >> 100% of the time >> >> Here are some implementation options that I *will* immediately kill: >> >> 1) Try to use the format to figure out record boundaries >> 2) Try to use the format to figure out the date of the record >> 3) Try to use the format to extract exception information >> 4) Any and all other format-based ideas not mentioned here >> >> The common theme is - it has to work 100% of the time. No excuses. >> >> On 10/08/2013 10:55 AM, James R. Perkins wrote: >>> On 10/08/2013 03:40 AM, Kabir Khan wrote: >>>> Just a thought regarding your PR >>>> https://github.com/wildfly/wildfly/pull/5156 >>>> >>>> The thing that stands out to me is that you're able to read the >>>> first 100 lines, e.g. >>>> :read-log-file(name=server.log, lines=100, tail=false) >>>> or the last 100 lines >>>> :read-log-file(name=server.log, lines=100, tail=true) >>>> >>>> Would it also be beneficial to also be allowed to start somewhere >>>> in the middle, e.g >>>> :read-log-file(name=server.log, lines=100, start=10235) >>> There is an option skip that will allows something similar. The >>> following would read read the last 100 lines _after_ 50 lines are >>> skipped from the bottom. So in a 200 line this would read lines 50 to >>> line 150. >>> :read-log-file(name=server.log, lines=100, skip=50, tail=true) >>> >>> The following would read from line 50 to line 150 >>> :read-log-file(name=server.log, lines=100, skip=50, >>> tail=false) >>> >>> I am using a BufferedReader so if an odd line terminator is used it >>> will >>> likely not work so well. I considered allowing for the line terminator >>> to be defined in the operation, but it would require analyzing >>> bytes for >>> the terminator and seemed like more work than it was worth. I would >>> guess the majority of users use the standard /n or /r/n terminators. >>>> or (19:38:01 today) >>>> :read-log-file(name=server.log, lines=100, >>>> start-time=20131008-10:38:01) >>> The date would be rather difficult. We're not parsing the log strings >>> just reading the raw text and sending it back. >>>> I'm not saying it MUST be like that but it might be worth discussing >>>> >>>> On 25 Sep 2013, at 16:36, Ondrej Zizka wrote: >>>> >>>>> Right, by "current" I meant to read the logging setup and if there's >>>>> just one file appender, take it as default. Or, if that would be >>>>> inappropriate mixing of abstraction layers, just "server.log". >>>>> >>>>> >>>>> On 25.9.2013 12:06, Kabir Khan wrote: >>>>>> Current log could be the server.log in a standard setup. If >>>>>> someone has done something more advanced in their logging setup, >>>>>> the name becomes necessary >>>>>> On 25 Sep 2013, at 08:34, Jaikiran Pai wrote: >>>>>> >>>>>>> On Wednesday 25 September 2013 09:25 AM, Ondrej Zizka wrote: >>>>>>>> 1) Could it have a "read-log-file()" without name= specified, >>>>>>>> which would read the "current" log file? >>>>>>> Given the way logger categories and appenders/handlers >>>>>>> interact, within a logging framework, I don't think there's any >>>>>>> notion of "current log file". It's a very valid scenario where >>>>>>> a single logging category can be backed by different appenders >>>>>>> (some of them file appenders) with different attributes and >>>>>>> each such appender writing out to a different file. So having a >>>>>>> name of the log file you want to view, becomes necessary. >>>>>>> >>>>>>> -Jaikiran >>>>>>> >>>>>>>> 2) Regarding security - what, besides logs, do we expect to be >>>>>>>> in the log dir? Could the admin block it by setting write-only >>>>>>>> rights? >>>>>>>> >>>>>>>> Ondra >>>>>>>> >>>>>>>> On 25.9.2013 02:40, James R. Perkins wrote: >>>>>>>>> I'm replying to this old thread to reopen this conversation >>>>>>>>> about reading log files. I've complete some work [1] on >>>>>>>>> reading log files via an operation. This is not exactly like >>>>>>>>> the JIRA suggests where it would only read the last 10 error >>>>>>>>> messages. All this change allows is the raw contents of the >>>>>>>>> file to be read. The idea is this could be used to read the >>>>>>>>> entire contents of the log file as a whole, or in chunks. >>>>>>>>> >>>>>>>>> What I've done is added two new operations list-log-files and >>>>>>>>> read-log-file. >>>>>>>>> >>>>>>>>> The list-log-files simply lists all files in the >>>>>>>>> jboss.server.log.dir. This may or may not be a good idea >>>>>>>>> really. I can see some potential security risks here mainly >>>>>>>>> just seeing files that may contain sensitive data. One way >>>>>>>>> I've thought of to get around that is read the logging >>>>>>>>> subsystem model and only show files from known types like the >>>>>>>>> file-handlers. The main issue with that is there is no good >>>>>>>>> way to get this to work for custom-handlers. >>>>>>>>> >>>>>>>>> The read-log-file simple does what it says and reads the >>>>>>>>> contents of a log file line by line. Reading line by line >>>>>>>>> should work for the most part unless the an non-standard line >>>>>>>>> delimiter is used. There are 5 options for this option; >>>>>>>>> • name (required): the name of the log file to read >>>>>>>>> • encoding: the encoding for the log file >>>>>>>>> • lines: the number of lines to read, defaults to 10 >>>>>>>>> • skip: the number of lines to skip before adding the >>>>>>>>> results >>>>>>>>> • tail: true to read from the bottom up, default is true >>>>>>>>> The result of this is just a list of lines with the \n or >>>>>>>>> \r\n stripped. Just to clarify too a line means a line in the >>>>>>>>> file, not a log record e.g. stack traces are generally >>>>>>>>> composed of multiple lines. >>>>>>>>> >>>>>>>>> So this begs the question, will this work for what we want? >>>>>>>>> What concerns does anyone else have? >>>>>>>>> >>>>>>>>> I have not yet submitted a PR yet as I wanted to get some >>>>>>>>> feedback before we bake it in. >>>>>>>>> >>>>>>>>> >>>>>>>>> [1]: https://github.com/jamezp/wildfly/compare/WFLY-280-read >>>>>>>>> >>>>>>>>> >>>>>>>>> On 08/14/2013 10:03 AM, James R. Perkins wrote: >>>>>>>>>> I had posted this to another list, but this is a more >>>>>>>>>> appropriate place for it. I think there needs to be a >>>>>>>>>> general discussion around this as it's been mentioned, at >>>>>>>>>> least to me, a few times here and there and I know Heiko >>>>>>>>>> raised the issue some time a go now. >>>>>>>>>> >>>>>>>>>> The original JIRA, WFLY-280[1], is to display the last 10 >>>>>>>>>> error messages only. To be honest I wouldn't find that very >>>>>>>>>> useful. To me if I'm looking for logs I want to see all >>>>>>>>>> logs, but that's not always so easy. Like the syslog-handler >>>>>>>>>> which doesn't log to a file so there is no way to read those >>>>>>>>>> messages back. >>>>>>>>>> >>>>>>>>>> The current plan for the last 10 error messages is we store >>>>>>>>>> messages in a queue that can be accessed via an operation. >>>>>>>>>> This works fine until the error message you're interested in >>>>>>>>>> is 11 or you want to see warning messages. >>>>>>>>>> >>>>>>>>>> Another option I had come up with is reading back the >>>>>>>>>> contents of the file, for example the server.log. This could >>>>>>>>>> be problematic too in that there is no way to filter >>>>>>>>>> information like only see error messages or only see warning >>>>>>>>>> messages. To solve this I have considered creating a JSON >>>>>>>>>> formatter so the results could be queried, but I don't think >>>>>>>>>> it should be a default which would mean it's not reliable >>>>>>>>>> for the console to assume it's getting back JSON. >>>>>>>>>> >>>>>>>>>> I've also thought about, haven't tested this and it may not >>>>>>>>>> work at all, creating a handler that uses websockets to send >>>>>>>>>> messages. I'm not sure how well this would work and it's >>>>>>>>>> possible it may not even work for bootstrap logging. >>>>>>>>>> >>>>>>>>>> With regards to audit logging, we're probably going to have >>>>>>>>>> to do something totally different from what we'll do in the >>>>>>>>>> logging subsystem since it doesn't use standard logging. >>>>>>>>>> >>>>>>>>>> I guess the bottom line is what does the console want to >>>>>>>>>> see? Do you want to see all raw text log messages? Do you >>>>>>>>>> want all messages but in a format like JSON that you can >>>>>>>>>> query/filter? Do you really want only the last 10 error >>>>>>>>>> messages only? All or none of these might be possible, but I >>>>>>>>>> really need to understand the needs before I can explore >>>>>>>>>> more in depth what the best option would be. >>>>>>>>>> >>>>>>>>>> [1]: https://issues.jboss.org/browse/WFLY-280 >>>>>>>>>> -- >>>>>>>>>> James R. Perkins >>>>>>>>>> Red Hat JBoss Middleware >>>>>>>>>> >>>>>>>>> -- >>>>>>>>> James R. Perkins >>>>>>>>> Red Hat JBoss Middleware >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> wildfly-dev mailing list >>>>>>>>> >>>>>>>>> wildfly-dev(a)lists.jboss.org >>>>>>>>> https://lists.jboss.org/mailman/listinfo/wildfly-dev >>>>>>>> _______________________________________________ >>>>>>>> wildfly-dev mailing list >>>>>>>> >>>>>>>> wildfly-dev(a)lists.jboss.org >>>>>>>> https://lists.jboss.org/mailman/listinfo/wildfly-dev >>>>>>> _______________________________________________ >>>>>>> wildfly-dev mailing list >>>>>>> wildfly-dev(a)lists.jboss.org >>>>>>> https://lists.jboss.org/mailman/listinfo/wildfly-dev >>>>> _______________________________________________ >>>>> wildfly-dev mailing list >>>>> wildfly-dev(a)lists.jboss.org >>>>> https://lists.jboss.org/mailman/listinfo/wildfly-dev

Is it possible to read log files not stored under ${jboss.server.log.dir}? I'm asking because the audit log is per default stored under ${jboss.server.data.dir} --- Harald Pehl JBoss by Red Hat http://hpehl.info Am 09.10.2013 um 18:10 schrieb James R. Perkins <jperkins(a)redhat.com <mailto:jperkins@redhat.com>>: > In my opinion we shouldn't offer any kind of log parsing options. We > should simple deliver the raw log file and have the end user use > their own parsing. There is a JIRA [1] to allow custom formatters to > be used. This means an end user could use an XML formatter and it > could be easily parsed by them. Since format patterns are so > configurable and we're going to expose the ability to override the > formatter we need to just give them the raw file. > > I suppose saying all that contradicts my reasoning for using the > BufferedReader.readLine() and maybe we should be returning by number > of bytes/chars instead of by line. There seems to be no real good or > solid answer and it would be nice to hear from any users that have an > opinion on how they might use it. From the web console I'm going to > guess either approach, by line or by bytes, will work fine as it > might just construct one large file for download. If using it from a > console just to see the last 20 lines of the log file, a line-by-line > approach to me makes more sense. > > [1]: https://issues.jboss.org/browse/WFLY-1188 > > On 10/09/2013 05:09 AM, Ondrej Zizka wrote: >> How about storing the offsets of the records. >> Then the operation would read this "index" and instead of >> format-based parsing, it would simply cut the right part. >> Simple, fast in both write and read, and doesn't consume too much >> memory/storage. >> >> Ondra >> >> >> On 8.10.2013 18:57, James R. Perkins wrote: >>> I'm definitely not trying to parse any of the log message format. I am >>> using a BufferedReader.readLine() which I was a little iffy on, but >>> went >>> with "lines" as opposed to reading x bytes. Obviously from a >>> management >>> standpoint reading the number of bytes makes more sense as we're not >>> relying on any kind of normal line terminator being used. From a user >>> standpoint though I think it makes less sense and the number of lines >>> makes more sense. >>> >>> I suppose we could have two operations so the web console could use >>> the >>> one that reads bytes and then have one a simple tail like one that >>> reads >>> by line. >>> >>> On 10/08/2013 09:26 AM, David M. Lloyd wrote: >>>> I've said it before and I'll say it again. Using the format to >>>> "intelligently" parse the logs looks like a good idea. It is >>>> actually a >>>> bad idea. A bad idea that looks like a good idea, or maybe even a >>>> "good >>>> enough" idea. Maybe it looks like an "easy" idea. But it's bad. >>>> Bad, >>>> and also not good. It will result in bug reports that nobody will >>>> bother fixing or implementing because they will be reasonable yet >>>> unfeasible to fix. >>>> >>>> Here are some implementation options that I *won't* immediately kill: >>>> >>>> 1) Treat the files in the log directory as opaque things that the >>>> server >>>> does not pretend to have interior knowledge of - in other words, >>>> provide >>>> operations that work on the file as a whole (or by things which make >>>> sense at a file level, like line number or dumb string grep) which >>>> will >>>> work 100% of the time >>>> 2) Provide in-memory (object) or on-disk (structured binary) >>>> storage of >>>> log record objects with intelligent search capabilities which will >>>> work >>>> 100% of the time >>>> >>>> Here are some implementation options that I *will* immediately kill: >>>> >>>> 1) Try to use the format to figure out record boundaries >>>> 2) Try to use the format to figure out the date of the record >>>> 3) Try to use the format to extract exception information >>>> 4) Any and all other format-based ideas not mentioned here >>>> >>>> The common theme is - it has to work 100% of the time. No excuses. >>>> >>>> On 10/08/2013 10:55 AM, James R. Perkins wrote: >>>>> On 10/08/2013 03:40 AM, Kabir Khan wrote: >>>>>> Just a thought regarding your PR >>>>>> https://github.com/wildfly/wildfly/pull/5156 >>>>>> >>>>>> The thing that stands out to me is that you're able to read the >>>>>> first 100 lines, e.g. >>>>>> :read-log-file(name=server.log, lines=100, tail=false) >>>>>> or the last 100 lines >>>>>> :read-log-file(name=server.log, lines=100, tail=true) >>>>>> >>>>>> Would it also be beneficial to also be allowed to start >>>>>> somewhere in the middle, e.g >>>>>> :read-log-file(name=server.log, lines=100, start=10235) >>>>> There is an option skip that will allows something similar. The >>>>> following would read read the last 100 lines _after_ 50 lines are >>>>> skipped from the bottom. So in a 200 line this would read lines >>>>> 50 to >>>>> line 150. >>>>> :read-log-file(name=server.log, lines=100, skip=50, >>>>> tail=true) >>>>> >>>>> The following would read from line 50 to line 150 >>>>> :read-log-file(name=server.log, lines=100, skip=50, >>>>> tail=false) >>>>> >>>>> I am using a BufferedReader so if an odd line terminator is used >>>>> it will >>>>> likely not work so well. I considered allowing for the line >>>>> terminator >>>>> to be defined in the operation, but it would require analyzing >>>>> bytes for >>>>> the terminator and seemed like more work than it was worth. I would >>>>> guess the majority of users use the standard /n or /r/n terminators. >>>>>> or (19:38:01 today) >>>>>> :read-log-file(name=server.log, lines=100, >>>>>> start-time=20131008-10:38:01) >>>>> The date would be rather difficult. We're not parsing the log >>>>> strings >>>>> just reading the raw text and sending it back. >>>>>> I'm not saying it MUST be like that but it might be worth >>>>>> discussing >>>>>> >>>>>> On 25 Sep 2013, at 16:36, Ondrej Zizka wrote: >>>>>> >>>>>>> Right, by "current" I meant to read the logging setup and if >>>>>>> there's >>>>>>> just one file appender, take it as default. Or, if that would be >>>>>>> inappropriate mixing of abstraction layers, just "server.log". >>>>>>> >>>>>>> >>>>>>> On 25.9.2013 12:06, Kabir Khan wrote: >>>>>>>> Current log could be the server.log in a standard setup. If >>>>>>>> someone has done something more advanced in their logging >>>>>>>> setup, the name becomes necessary >>>>>>>> On 25 Sep 2013, at 08:34, Jaikiran Pai wrote: >>>>>>>> >>>>>>>>> On Wednesday 25 September 2013 09:25 AM, Ondrej Zizka wrote: >>>>>>>>>> 1) Could it have a "read-log-file()" without name= >>>>>>>>>> specified, which would read the "current" log file? >>>>>>>>> Given the way logger categories and appenders/handlers >>>>>>>>> interact, within a logging framework, I don't think there's >>>>>>>>> any notion of "current log file". It's a very valid scenario >>>>>>>>> where a single logging category can be backed by different >>>>>>>>> appenders (some of them file appenders) with different >>>>>>>>> attributes and each such appender writing out to a different >>>>>>>>> file. So having a name of the log file you want to view, >>>>>>>>> becomes necessary. >>>>>>>>> >>>>>>>>> -Jaikiran >>>>>>>>> >>>>>>>>>> 2) Regarding security - what, besides logs, do we expect to >>>>>>>>>> be in the log dir? Could the admin block it by setting >>>>>>>>>> write-only rights? >>>>>>>>>> >>>>>>>>>> Ondra >>>>>>>>>> >>>>>>>>>> On 25.9.2013 02:40, James R. Perkins wrote: >>>>>>>>>>> I'm replying to this old thread to reopen this conversation >>>>>>>>>>> about reading log files. I've complete some work [1] on >>>>>>>>>>> reading log files via an operation. This is not exactly >>>>>>>>>>> like the JIRA suggests where it would only read the last 10 >>>>>>>>>>> error messages. All this change allows is the raw contents >>>>>>>>>>> of the file to be read. The idea is this could be used to >>>>>>>>>>> read the entire contents of the log file as a whole, or in >>>>>>>>>>> chunks. >>>>>>>>>>> >>>>>>>>>>> What I've done is added two new operations list-log-files >>>>>>>>>>> and read-log-file. >>>>>>>>>>> >>>>>>>>>>> The list-log-files simply lists all files in the >>>>>>>>>>> jboss.server.log.dir. This may or may not be a good idea >>>>>>>>>>> really. I can see some potential security risks here mainly >>>>>>>>>>> just seeing files that may contain sensitive data. One way >>>>>>>>>>> I've thought of to get around that is read the logging >>>>>>>>>>> subsystem model and only show files from known types like >>>>>>>>>>> the file-handlers. The main issue with that is there is no >>>>>>>>>>> good way to get this to work for custom-handlers. >>>>>>>>>>> >>>>>>>>>>> The read-log-file simple does what it says and reads the >>>>>>>>>>> contents of a log file line by line. Reading line by line >>>>>>>>>>> should work for the most part unless the an non-standard >>>>>>>>>>> line delimiter is used. There are 5 options for this option; >>>>>>>>>>> • name (required): the name of the log file to read >>>>>>>>>>> • encoding: the encoding for the log file >>>>>>>>>>> • lines: the number of lines to read, defaults to 10 >>>>>>>>>>> • skip: the number of lines to skip before adding the >>>>>>>>>>> results >>>>>>>>>>> • tail: true to read from the bottom up, default is true >>>>>>>>>>> The result of this is just a list of lines with the \n or >>>>>>>>>>> \r\n stripped. Just to clarify too a line means a line in >>>>>>>>>>> the file, not a log record e.g. stack traces are generally >>>>>>>>>>> composed of multiple lines. >>>>>>>>>>> >>>>>>>>>>> So this begs the question, will this work for what we want? >>>>>>>>>>> What concerns does anyone else have? >>>>>>>>>>> >>>>>>>>>>> I have not yet submitted a PR yet as I wanted to get some >>>>>>>>>>> feedback before we bake it in. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> [1]: https://github.com/jamezp/wildfly/compare/WFLY-280-read >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On 08/14/2013 10:03 AM, James R. Perkins wrote: >>>>>>>>>>>> I had posted this to another list, but this is a more >>>>>>>>>>>> appropriate place for it. I think there needs to be a >>>>>>>>>>>> general discussion around this as it's been mentioned, at >>>>>>>>>>>> least to me, a few times here and there and I know Heiko >>>>>>>>>>>> raised the issue some time a go now. >>>>>>>>>>>> >>>>>>>>>>>> The original JIRA, WFLY-280[1], is to display the last 10 >>>>>>>>>>>> error messages only. To be honest I wouldn't find that >>>>>>>>>>>> very useful. To me if I'm looking for logs I want to see >>>>>>>>>>>> all logs, but that's not always so easy. Like the >>>>>>>>>>>> syslog-handler which doesn't log to a file so there is no >>>>>>>>>>>> way to read those messages back. >>>>>>>>>>>> >>>>>>>>>>>> The current plan for the last 10 error messages is we >>>>>>>>>>>> store messages in a queue that can be accessed via an >>>>>>>>>>>> operation. This works fine until the error message you're >>>>>>>>>>>> interested in is 11 or you want to see warning messages. >>>>>>>>>>>> >>>>>>>>>>>> Another option I had come up with is reading back the >>>>>>>>>>>> contents of the file, for example the server.log. This >>>>>>>>>>>> could be problematic too in that there is no way to filter >>>>>>>>>>>> information like only see error messages or only see >>>>>>>>>>>> warning messages. To solve this I have considered creating >>>>>>>>>>>> a JSON formatter so the results could be queried, but I >>>>>>>>>>>> don't think it should be a default which would mean it's >>>>>>>>>>>> not reliable for the console to assume it's getting back >>>>>>>>>>>> JSON. >>>>>>>>>>>> >>>>>>>>>>>> I've also thought about, haven't tested this and it may >>>>>>>>>>>> not work at all, creating a handler that uses websockets >>>>>>>>>>>> to send messages. I'm not sure how well this would work >>>>>>>>>>>> and it's possible it may not even work for bootstrap logging. >>>>>>>>>>>> >>>>>>>>>>>> With regards to audit logging, we're probably going to >>>>>>>>>>>> have to do something totally different from what we'll do >>>>>>>>>>>> in the logging subsystem since it doesn't use standard >>>>>>>>>>>> logging. >>>>>>>>>>>> >>>>>>>>>>>> I guess the bottom line is what does the console want to >>>>>>>>>>>> see? Do you want to see all raw text log messages? Do you >>>>>>>>>>>> want all messages but in a format like JSON that you can >>>>>>>>>>>> query/filter? Do you really want only the last 10 error >>>>>>>>>>>> messages only? All or none of these might be possible, but >>>>>>>>>>>> I really need to understand the needs before I can explore >>>>>>>>>>>> more in depth what the best option would be. >>>>>>>>>>>> >>>>>>>>>>>> [1]: https://issues.jboss.org/browse/WFLY-280 >>>>>>>>>>>> -- >>>>>>>>>>>> James R. Perkins >>>>>>>>>>>> Red Hat JBoss Middleware >>>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> James R. Perkins >>>>>>>>>>> Red Hat JBoss Middleware >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> wildfly-dev mailing list >>>>>>>>>>> >>>>>>>>>>> wildfly-dev(a)lists.jboss.org >>>>>>>>>>> https://lists.jboss.org/mailman/listinfo/wildfly-dev >>>>>>>>>> _______________________________________________ >>>>>>>>>> wildfly-dev mailing list >>>>>>>>>> >>>>>>>>>> wildfly-dev(a)lists.jboss.org >>>>>>>>>> https://lists.jboss.org/mailman/listinfo/wildfly-dev >>>>>>>>> _______________________________________________ >>>>>>>>> wildfly-dev mailing list >>>>>>>>> wildfly-dev(a)lists.jboss.org >>>>>>>>> https://lists.jboss.org/mailman/listinfo/wildfly-dev >>>>>>> _______________________________________________ >>>>>>> wildfly-dev mailing list >>>>>>> wildfly-dev(a)lists.jboss.org >>>>>>> https://lists.jboss.org/mailman/listinfo/wildfly-dev >> > > -- > James R. Perkins > Red Hat JBoss Middleware > _______________________________________________ > wildfly-dev mailing list > wildfly-dev(a)lists.jboss.org <mailto:wildfly-dev@lists.jboss.org> > https://lists.jboss.org/mailman/listinfo/wildfly-dev

1) Could it have a "read-log-file()" without name= specified, which would read the "current" log file?

2) Regarding security - what, besides logs, do we expect to be in the log dir? Could the admin block it by setting write-only rights?

Ondra On 25.9.2013 02:40, James R. Perkins wrote: > I'm replying to this old thread to reopen this conversation about > reading log files. I've complete some work [1] on reading log files > via an operation. This is not exactly like the JIRA suggests where it > would only read the last 10 error messages. All this change allows is > the raw contents of the file to be read. The idea is this could be > used to read the entire contents of the log file as a whole, or in > chunks. > > What I've done is added two new operations list-log-files and > read-log-file. > > The list-log-files simply lists all files in the > jboss.server.log.dir. This may or may not be a good idea really. I > can see some potential security risks here mainly just seeing files > that may contain sensitive data. One way I've thought of to get > around that is read the logging subsystem model and only show files > from known types like the file-handlers. The main issue with that is > there is no good way to get this to work for custom-handlers. > > The read-log-file simple does what it says and reads the contents of > a log file line by line. Reading line by line should work for the > most part unless the an non-standard line delimiter is used. There > are 5 options for this option; > > * name (required): the name of the log file to read > * encoding: the encoding for the log file > * lines: the number of lines to read, defaults to 10 > * skip: the number of lines to skip before adding the results > * tail: true to read from the bottom up, default is true > > The result of this is just a list of lines with the \n or \r\n > stripped. Just to clarify too a line means a line in the file, not a > log record e.g. stack traces are generally composed of multiple lines. > > So this begs the question, will this work for what we want? What > concerns does anyone else have? > > I have not yet submitted a PR yet as I wanted to get some feedback > before we bake it in. > > > [1]: https://github.com/jamezp/wildfly/compare/WFLY-280-read > > > On 08/14/2013 10:03 AM, James R. Perkins wrote: >> I had posted this to another list, but this is a more appropriate >> place for it. I think there needs to be a general discussion around >> this as it's been mentioned, at least to me, a few times here and >> there and I know Heiko raised the issue some time a go now. >> >> The original JIRA, WFLY-280[1], is to display the last 10 error >> messages only. To be honest I wouldn't find that very useful. To me >> if I'm looking for logs I want to see all logs, but that's not >> always so easy. Like the syslog-handler which doesn't log to a file >> so there is no way to read those messages back. >> >> The current plan for the last 10 error messages is we store messages >> in a queue that can be accessed via an operation. This works fine >> until the error message you're interested in is 11 or you want to >> see warning messages. >> >> Another option I had come up with is reading back the contents of >> the file, for example the server.log. This could be problematic too >> in that there is no way to filter information like only see error >> messages or only see warning messages. To solve this I have >> considered creating a JSON formatter so the results could be >> queried, but I don't think it should be a default which would mean >> it's not reliable for the console to assume it's getting back JSON. >> >> I've also thought about, haven't tested this and it may not work at >> all, creating a handler that uses websockets to send messages. I'm >> not sure how well this would work and it's possible it may not even >> work for bootstrap logging. >> >> With regards to audit logging, we're probably going to have to do >> something totally different from what we'll do in the logging >> subsystem since it doesn't use standard logging. >> >> I guess the bottom line is what does the console want to see? Do you >> want to see all raw text log messages? Do you want all messages but >> in a format like JSON that you can query/filter? Do you really want >> only the last 10 error messages only? All or none of these might be >> possible, but I really need to understand the needs before I can >> explore more in depth what the best option would be. >> >> [1]: https://issues.jboss.org/browse/WFLY-280 >> -- >> James R. Perkins >> Red Hat JBoss Middleware > > -- > James R. Perkins > Red Hat JBoss Middleware > > > _______________________________________________ > wildfly-dev mailing list > wildfly-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/wildfly-dev