1:57 p.m.
Hi guys,
in order to keep track of notable events (nodes joining, leaving,
rebalances, availability, task execution) and present them in the
console I was thinking of using specific logging categories. These logs
would be then collected by dedicated appenders on each node. The idea
would then be to have some distexec tasks exposed to the management
interface (console/CLI) which can query and aggregate the data.
Initially I was thinking about using a local cache per-node with a SFCS,
eviction, expiration and indexing and creating a log4j appender which
writes to this cache, but probably a simpler lucene-based appender would
suffice.
WDYT ?
Tristan
--
Tristan Tarrant
Infinispan Lead
JBoss, a division of Red Hat
2:32 p.m.
I really like the idea!
Would it be possible to forward raw data (not aggregated) to the ELK stack
[1]?
This aspect might be important for production systems with centralized
logging.
[1] http://wildfly.org/news/2015/07/25/Wildfly-And-ELK/
On Fri, Nov 13, 2015 at 1:57 PM, Tristan Tarrant <ttarrant(a)redhat.com>
wrote:
Hi guys,
in order to keep track of notable events (nodes joining, leaving,
rebalances, availability, task execution) and present them in the
console I was thinking of using specific logging categories. These logs
would be then collected by dedicated appenders on each node. The idea
would then be to have some distexec tasks exposed to the management
interface (console/CLI) which can query and aggregate the data.
Initially I was thinking about using a local cache per-node with a SFCS,
eviction, expiration and indexing and creating a log4j appender which
writes to this cache, but probably a simpler lucene-based appender would
suffice.
WDYT ?
Tristan
--
Tristan Tarrant
Infinispan Lead
JBoss, a division of Red Hat
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
2:42 p.m.
The Hibernate Search project is now working on giving end users the
option to write to either
- Lucene (embedded/local and optionally into Infinispan's Lucene Directory)
- Elastic Search -> would give you ELK integration
- [later] Solr
We have a working prototype, and just yesterday some of the first
steps were merged in master.
This implies that if you write those events into an ad-hoc Infinispan
Cache, you inherit the functionality via Infinispan Query and benefit
from the decoupling and pluggability of our indexing components via a
stable API.
I'd suggest you just write those events into a dedicated Cache; you
could index the cache by default and I'm happy to help defining an
appropriate "domain model" to record such events..
Clearly indexing such a cache has benefits, but assuming you don't
need some Lucene-only statistics you'd have the option for non-indexed
queries and in the near future for ELK connectors.
Sanne
On 13 November 2015 at 13:32, Sebastian Laskawiec <slaskawi(a)redhat.com> wrote:
I really like the idea!
Would it be possible to forward raw data (not aggregated) to the ELK stack
[1]?
This aspect might be important for production systems with centralized
logging.
[1] http://wildfly.org/news/2015/07/25/Wildfly-And-ELK/
On Fri, Nov 13, 2015 at 1:57 PM, Tristan Tarrant <ttarrant(a)redhat.com>
wrote:
>
> Hi guys,
>
> in order to keep track of notable events (nodes joining, leaving,
> rebalances, availability, task execution) and present them in the
> console I was thinking of using specific logging categories. These logs
> would be then collected by dedicated appenders on each node. The idea
> would then be to have some distexec tasks exposed to the management
> interface (console/CLI) which can query and aggregate the data.
> Initially I was thinking about using a local cache per-node with a SFCS,
> eviction, expiration and indexing and creating a log4j appender which
> writes to this cache, but probably a simpler lucene-based appender would
> suffice.
>
> WDYT ?
>
> Tristan
> --
> Tristan Tarrant
> Infinispan Lead
> JBoss, a division of Red Hat
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
2:50 p.m.
On 13/11/2015 14:42, Sanne Grinovero wrote:
The Hibernate Search project is now working on giving end users the
option to write to either
- Lucene (embedded/local and optionally into Infinispan's Lucene Directory)
- Elastic Search -> would give you ELK integration
- [later] Solr
Nice !
I'd suggest you just write those events into a dedicated Cache;
you
could index the cache by default and I'm happy to help defining an
appropriate "domain model" to record such events..
Ok.
Obviously the cache would not be clustered: we still need to simulate
clustered queries on top of all the nodes using distexec.
Tristan
--
Tristan Tarrant
Infinispan Lead
JBoss, a division of Red Hat
2:53 p.m.
On Fri, Nov 13, 2015 at 12:57 PM, Tristan Tarrant <ttarrant(a)redhat.com>
wrote:
Hi guys,
in order to keep track of notable events (nodes joining, leaving,
rebalances, availability, task execution) and present them in the
console I was thinking of using specific logging categories. These logs
would be then collected by dedicated appenders on each node. The idea
would then be to have some distexec tasks exposed to the management
interface (console/CLI) which can query and aggregate the data.
Initially I was thinking about using a local cache per-node with a SFCS,
eviction, expiration and indexing and creating a log4j appender which
writes to this cache, but probably a simpler lucene-based appender would
suffice.
I'd favor using an indexed cache instead simple Lucene appenders, since
they
take care of querying, indexing and after all they expose the Lucene API if
needed.
Not sure about local caches, what about log retention when the node dies?
Gustavo
WDYT ?
Tristan
--
Tristan Tarrant
Infinispan Lead
JBoss, a division of Red Hat
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
3:11 p.m.
On 13/11/2015 14:53, Gustavo Fernandes wrote:
Not sure about local caches, what about log retention when the node
dies?
I wanted a solution which can suffer splits and not adversely affect the
cluster otherwise. But I'm open for counterarguments.
Tristan
--
Tristan Tarrant
Infinispan Lead
JBoss, a division of Red Hat
3:27 p.m.
On 13 November 2015 at 14:11, Tristan Tarrant <ttarrant(a)redhat.com> wrote:
On 13/11/2015 14:53, Gustavo Fernandes wrote:
> Not sure about local caches, what about log retention when the node dies?
I wanted a solution which can suffer splits and not adversely affect the
cluster otherwise. But I'm open for counterarguments.
The indexer can be a service which lives independently from
Infinispan; ELK would be one of the options, but there are others
already (JGroups, JMS).
But obviously you'd have a problem when the grid is fine, but there's
a split between the Infinispan grid and such service..
configuring an embedded local Lucene indexer would be better.
When using the JMS backend we have the option to store the logs
locally and replicate them to ELK when the connection to ELK is
restored; you'd need JMS for that one to work, but it should be easy
to implement a similar thing using a simplified disk journal. I guess
what I'm staying is that the Search back-end is pluggable, and having
one which tries remote first or logs to disk otherwise should be easy
(and a welcome reusable backend for other circumstances).
Sanne
Tristan
--
Tristan Tarrant
Infinispan Lead
JBoss, a division of Red Hat
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
3:53 p.m.
On Fri, Nov 13, 2015 at 4:27 PM, Sanne Grinovero <sanne(a)infinispan.org> wrote:
On 13 November 2015 at 14:11, Tristan Tarrant
<ttarrant(a)redhat.com> wrote:
> On 13/11/2015 14:53, Gustavo Fernandes wrote:
>> Not sure about local caches, what about log retention when the node dies?
>
> I wanted a solution which can suffer splits and not adversely affect the
> cluster otherwise. But I'm open for counterarguments.
The indexer can be a service which lives independently from
Infinispan; ELK would be one of the options, but there are others
already (JGroups, JMS).
But obviously you'd have a problem when the grid is fine, but there's
a split between the Infinispan grid and such service..
configuring an embedded local Lucene indexer would be better.
When using the JMS backend we have the option to store the logs
locally and replicate them to ELK when the connection to ELK is
restored; you'd need JMS for that one to work, but it should be easy
to implement a similar thing using a simplified disk journal. I guess
what I'm staying is that the Search back-end is pluggable, and having
one which tries remote first or logs to disk otherwise should be easy
(and a welcome reusable backend for other circumstances).
+1 to use a local cache by default, and maybe support additional
(async) replication to another service.
Tristan, can the users configure WildFly/Infinispan Server to use
another logging service instead of Log4J2? If yes, perhaps
implementing this functionality with custom appenders is not a good
idea...
Cheers
Dan
4:28 p.m.
On 13/11/2015 15:53, Dan Berindei wrote:
Tristan, can the users configure WildFly/Infinispan Server to use
another logging service instead of Log4J2? If yes, perhaps
implementing this functionality with custom appenders is not a good
idea...
The WildFly logging goes through the lgoging subsystem which is built on
top of jboss-logging with direct support for wrapping to log4j 1.2.x
(not log4j2). You can plugin additional handlers [1], but you cannot
really change the underlying implementation. I'll ping James Perkins and
ask him about the details.
Tristan
[1] http://wildfly.org/news/2015/07/25/Wildfly-And-ELK/
--
Tristan Tarrant
Infinispan Lead
JBoss, a division of Red Hat
6:02 p.m.
On 13 November 2015 at 15:28, Tristan Tarrant <ttarrant(a)redhat.com> wrote:
On 13/11/2015 15:53, Dan Berindei wrote:
>
>
> Tristan, can the users configure WildFly/Infinispan Server to use
> another logging service instead of Log4J2? If yes, perhaps
> implementing this functionality with custom appenders is not a good
> idea...
The WildFly logging goes through the lgoging subsystem which is built on
top of jboss-logging with direct support for wrapping to log4j 1.2.x
(not log4j2). You can plugin additional handlers [1], but you cannot
really change the underlying implementation. I'll ping James Perkins and
ask him about the details.
Why bind this to log4j(2) ? Just write a JBoss-Logging backend, it's
easier and can wrap any other actual backend.
I've created one to test what specific stuff would be logged in the
Hibernate ORM testsuite; it's quite nice and easy as JBoss Logging
picks up extension points from classpath via the ServiceLoader
pattern.
Although, I'm surprised that you want to use an actual logging
category. The API for that is awful for rich events; I do realize that
"text" is a rather standard API, and stuff like ELK are great because
they can analyze it, but their goal is to consume any Log format,
while you are producing events so you can choose for a richer model.
Structure entities, annotate them and dump them in a Cache. You'll get
benefits like fields being separated already without having to regex
them on the consumer side, and things like numbers being mapped
correctly into optimal-for-range queries. It's always possible to
transform to text at the end if you want to.. just add a
toString()-like contract to your same entity so we could optionally
dump them to a standard appender as well, but don't make it your
primary API for logging rich events.
Sanne
Tristan
[1] http://wildfly.org/news/2015/07/25/Wildfly-And-ELK/
--
Tristan Tarrant
Infinispan Lead
JBoss, a division of Red Hat
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
3:04 p.m.
On 13 Nov 2015, at 18:02, Sanne Grinovero
<sanne(a)infinispan.org> wrote:
Although, I'm surprised that you want to use an actual logging
category. The API for that is awful for rich events; I do realize that
"text" is a rather standard API, and stuff like ELK are great because
they can analyze it, but their goal is to consume any Log format,
while you are producing events so you can choose for a richer model.
The L side of thing is all about taking the flat text and re-extract structure out of it
for proper indexing. So if you can avoid losing the structure in the first place, that’s
better.
7:02 p.m.
On 16 nov. 2015, at 15:04, Emmanuel Bernard
<emmanuel(a)hibernate.org> wrote:
> On 13 Nov 2015, at 18:02, Sanne Grinovero <sanne(a)infinispan.org> wrote:
>
>
> Although, I'm surprised that you want to use an actual logging
> category. The API for that is awful for rich events; I do realize that
> "text" is a rather standard API, and stuff like ELK are great because
> they can analyze it, but their goal is to consume any Log format,
> while you are producing events so you can choose for a richer model.
The L side of thing is all about taking the flat text and re-extract structure out of it
for proper indexing. So if you can avoid losing the structure in the first place, that’s
better.
L in ELK
3337
days inactive
3340
days old
infinispan-dev@lists.jboss.org
11 comments
6 participants
participants (6)
-
Dan Berindei -
Emmanuel Bernard -
Gustavo Fernandes -
Sanne Grinovero -
Sebastian Laskawiec -
Tristan Tarrant