In my opinion we
shouldn't offer any kind of log parsing options. We should
simple deliver the raw log file and have the end user use
their own parsing. There is a JIRA [1] to allow custom
formatters to be used. This means an end user could use an
XML formatter and it could be easily parsed by them. Since
format patterns are so configurable and we're going to
expose the ability to override the formatter we need to
just give them the raw file.
I suppose saying all that contradicts my reasoning for
using the BufferedReader.readLine() and maybe we should be
returning by number of bytes/chars instead of by line.
There seems to be no real good or solid answer and it
would be nice to hear from any users that have an opinion
on how they might use it. From the web console I'm going
to guess either approach, by line or by bytes, will work
fine as it might just construct one large file for
download. If using it from a console just to see the last
20 lines of the log file, a line-by-line approach to me
makes more sense.
[1]:
https://issues.jboss.org/browse/WFLY-1188
On 10/09/2013 05:09 AM,
Ondrej Zizka wrote:
How about storing the offsets of the
records.
Then the operation would read this "index" and instead
of format-based parsing, it would simply cut the right
part.
Simple, fast in both write and read, and doesn't consume
too much memory/storage.
Ondra
On 8.10.2013 18:57, James R. Perkins wrote:
I'm definitely not trying to
parse any of the log message format. I am
using a BufferedReader.readLine() which I was a little
iffy on, but went
with "lines" as opposed to reading x bytes. Obviously
from a management
standpoint reading the number of bytes makes more
sense as we're not
relying on any kind of normal line terminator being
used. From a user
standpoint though I think it makes less sense and the
number of lines
makes more sense.
I suppose we could have two operations so the web
console could use the
one that reads bytes and then have one a simple tail
like one that reads
by line.
On 10/08/2013 09:26 AM, David M. Lloyd wrote:
I've said it before and I'll
say it again. Using the format to
"intelligently" parse the logs looks like a good
idea. It is actually a
bad idea. A bad idea that looks like a good idea,
or maybe even a "good
enough" idea. Maybe it looks like an "easy" idea.
But it's bad. Bad,
and also not good. It will result in bug reports
that nobody will
bother fixing or implementing because they will be
reasonable yet
unfeasible to fix.
Here are some implementation options that I *won't*
immediately kill:
1) Treat the files in the log directory as opaque
things that the server
does not pretend to have interior knowledge of - in
other words, provide
operations that work on the file as a whole (or by
things which make
sense at a file level, like line number or dumb
string grep) which will
work 100% of the time
2) Provide in-memory (object) or on-disk (structured
binary) storage of
log record objects with intelligent search
capabilities which will work
100% of the time
Here are some implementation options that I *will*
immediately kill:
1) Try to use the format to figure out record
boundaries
2) Try to use the format to figure out the date of
the record
3) Try to use the format to extract exception
information
4) Any and all other format-based ideas not
mentioned here
The common theme is - it has to work 100% of the
time. No excuses.
On 10/08/2013 10:55 AM, James R. Perkins wrote:
On 10/08/2013 03:40 AM,
Kabir Khan wrote:
Just a thought regarding
your PR https://github.com/wildfly/wildfly/pull/5156
The thing that stands out to me is that you're
able to read the first 100 lines, e.g.
:read-log-file(name=server.log,
lines=100, tail=false)
or the last 100 lines
:read-log-file(name=server.log,
lines=100, tail=true)
Would it also be beneficial to also be allowed
to start somewhere in the middle, e.g
:read-log-file(name=server.log,
lines=100, start=10235)
There is an option skip that will allows something
similar. The
following would read read the last 100 lines
_after_ 50 lines are
skipped from the bottom. So in a 200 line this
would read lines 50 to
line 150.
:read-log-file(name=server.log, lines=100,
skip=50, tail=true)
The following would read from line 50 to line 150
:read-log-file(name=server.log, lines=100,
skip=50, tail=false)
I am using a BufferedReader so if an odd line
terminator is used it will
likely not work so well. I considered allowing for
the line terminator
to be defined in the operation, but it would
require analyzing bytes for
the terminator and seemed like more work than it
was worth. I would
guess the majority of users use the standard /n or
/r/n terminators.
or (19:38:01 today)
:read-log-file(name=server.log,
lines=100, start-time=20131008-10:38:01)
The date would be rather difficult. We're not
parsing the log strings
just reading the raw text and sending it back.
I'm not saying it MUST be
like that but it might be worth discussing
On 25 Sep 2013, at 16:36, Ondrej Zizka wrote:
Right, by "current" I
meant to read the logging setup and if there's
just one file appender, take it as default.
Or, if that would be
inappropriate mixing of abstraction layers,
just "server.log".
On 25.9.2013 12:06, Kabir Khan wrote:
Current log could be
the server.log in a standard setup. If
someone has done something more advanced in
their logging setup, the name becomes
necessary
On 25 Sep 2013, at 08:34, Jaikiran Pai
wrote:
On Wednesday 25
September 2013 09:25 AM, Ondrej Zizka
wrote:
1) Could it have a
"read-log-file()" without name=
specified, which would read the
"current" log file?
Given the way logger categories and
appenders/handlers interact, within a
logging framework, I don't think there's
any notion of "current log file". It's a
very valid scenario where a single logging
category can be backed by different
appenders (some of them file appenders)
with different attributes and each such
appender writing out to a different file.
So having a name of the log file you want
to view, becomes necessary.
-Jaikiran
2) Regarding
security - what, besides logs, do we
expect to be in the log dir? Could the
admin block it by setting write-only
rights?
Ondra
On 25.9.2013 02:40, James R. Perkins
wrote:
I'm replying to
this old thread to reopen this
conversation about reading log files.
I've complete some work [1] on reading
log files via an operation. This is
not exactly like the JIRA suggests
where it would only read the last 10
error messages. All this change allows
is the raw contents of the file to be
read. The idea is this could be used
to read the entire contents of the log
file as a whole, or in chunks.
What I've done is added two new
operations list-log-files and
read-log-file.
The list-log-files simply lists all
files in the jboss.server.log.dir.
This may or may not be a good idea
really. I can see some potential
security risks here mainly just seeing
files that may contain sensitive data.
One way I've thought of to get around
that is read the logging subsystem
model and only show files from known
types like the file-handlers. The main
issue with that is there is no good
way to get this to work for
custom-handlers.
The read-log-file simple does what it
says and reads the contents of a log
file line by line. Reading line by
line should work for the most part
unless the an non-standard line
delimiter is used. There are 5 options
for this option;
• name (required): the name of the
log file to read
• encoding: the encoding for the
log file
• lines: the number of lines to
read, defaults to 10
• skip: the number of lines to
skip before adding the results
• tail: true to read from the
bottom up, default is true
The result of this is just a list of
lines with the \n or \r\n stripped.
Just to clarify too a line means a
line in the file, not a log record
e.g. stack traces are generally
composed of multiple lines.
So this begs the question, will this
work for what we want? What concerns
does anyone else have?
I have not yet submitted a PR yet as I
wanted to get some feedback before we
bake it in.
[1]: https://github.com/jamezp/wildfly/compare/WFLY-280-read
On 08/14/2013 10:03 AM, James R.
Perkins wrote:
I had posted
this to another list, but this is a
more appropriate place for it. I
think there needs to be a general
discussion around this as it's been
mentioned, at least to me, a few
times here and there and I know
Heiko raised the issue some time a
go now.
The original JIRA, WFLY-280[1], is
to display the last 10 error
messages only. To be honest I
wouldn't find that very useful. To
me if I'm looking for logs I want to
see all logs, but that's not always
so easy. Like the syslog-handler
which doesn't log to a file so there
is no way to read those messages
back.
The current plan for the last 10
error messages is we store messages
in a queue that can be accessed via
an operation. This works fine until
the error message you're interested
in is 11 or you want to see warning
messages.
Another option I had come up with is
reading back the contents of the
file, for example the server.log.
This could be problematic too in
that there is no way to filter
information like only see error
messages or only see warning
messages. To solve this I have
considered creating a JSON formatter
so the results could be queried, but
I don't think it should be a default
which would mean it's not reliable
for the console to assume it's
getting back JSON.
I've also thought about, haven't
tested this and it may not work at
all, creating a handler that uses
websockets to send messages. I'm not
sure how well this would work and
it's possible it may not even work
for bootstrap logging.
With regards to audit logging, we're
probably going to have to do
something totally different from
what we'll do in the logging
subsystem since it doesn't use
standard logging.
I guess the bottom line is what does
the console want to see? Do you want
to see all raw text log messages? Do
you want all messages but in a
format like JSON that you can
query/filter? Do you really want
only the last 10 error messages
only? All or none of these might be
possible, but I really need to
understand the needs before I can
explore more in depth what the best
option would be.
[1]: https://issues.jboss.org/browse/WFLY-280
--
James R. Perkins
Red Hat JBoss Middleware
--
James R. Perkins
Red Hat JBoss Middleware
_______________________________________________
wildfly-dev mailing list
wildfly-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/wildfly-dev
_______________________________________________
wildfly-dev mailing list
wildfly-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/wildfly-dev
_______________________________________________
wildfly-dev mailing list
wildfly-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/wildfly-dev
_______________________________________________
wildfly-dev mailing list
wildfly-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/wildfly-dev
--
James R. Perkins
Red Hat JBoss Middleware