+1000 looks very interesting!
I think we should a nice visualization app for our conference
presentations... We might consider refreshing Visual [5], but personally I
would love to see something new. A NodeJS client might be also a nice way
to go but tracing how the data flows through the cluster... this would be
something nice...
Thanks
Sebastian
[5]
On Mon, Jun 6, 2016 at 3:39 PM, Galder Zamarreño <galder(a)redhat.com> wrote:
Hey guys,
Sorry for the delay. I somehow read the replies but forgot to reply :)
I think the target of Infispector and Zipkin based tracing would be
different, although there is some common ground. I think both Infispector
and Zipkin would be helpful in helping diagnose cluster performance issues
and get a general overview of how messages pass from one node to the other.
In terms of difference Infispector seems more geared towards education/qe
whereas Zipkin is more targeted as something we can run at runtime in
production for our users.
Although both projects have different targets I think we'll be able to
take advantage of each others work, e.g. custom JGroups protocols points,
byteman/bytebuddy rules for intercept scenarions...etc.
Great work leading all the Infispector work Tomas, looking forward to
demos/videos :D
Cheers,
--
Galder Zamarreño
Infinispan, Red Hat
> On 10 May 2016, at 15:53, Radim Vansa <rvansa(a)redhat.com> wrote:
>
> To complement this, MFT is a tool that won't offer any sleek charts or
> visualisations. It's tricky to use and understand - it's intended for
> developers as a tool for problem analysis. But it gets more in depth
> than InfiSpector, linking the information from different nodes, JFR
> events and so forth.
>
> R.
>
> On 05/10/2016 09:41 AM, Tomas Sykora wrote:
>> Hello Galder, and all!
>> It’s nice to communicate again via infinispan-dev after a while :)
>>
>> TL;DR: I can see some intersections with zipkin.io initiative goals
but InfiSpector seems to be much more “easier to handle and contribute to
at this moment” -- that suits more our student-related use case. Let’s
continue with the discussion :)
>>
>> Firstly, a short introduction into the context. Red Hat is running
Research & Development laboratory here in Brno at 2 biggest local
universities: Masaryk University, Faculty of Informatics (FI MU) and Brno
University of Technology, Faculty of Information Technologies (FIT VUT).
>> The aim is to better and sooner reach out to students, get them
involved into interesting project, show them open-source, git, git
workflows and many other things (project specific). An year ago I got
excited about this idea and started to think whether I can deliver such a
project. And I did.
>>
>> Team faces one big challenge and this is a time constraint. Students
are working on _several_ projects during their studies to fulfill courses’
requirements to pass the semester. It’s hard for them to find additional
time to be coding even something else. Team managed that but slowly, that’s
understandable though. Designing InfiSpector infrastructure took us some
time (Kafka, Druid, NodeJS) + evaluation of these technologies + proof of
concepts.
>>
>> All 5 team members are 2nd year students of bachelor studies at FIT VUT
Brno.
>> Marek Ciz (
https://github.com/mciz), also my very good friend from my
home town :) His primary domain is Druid, Kafka and infrastructure.
>> Vratislav Hais (
https://github.com/vratislavhais) Primary domain is
front-end.
>> Jan Fitz (
https://github.com/janfitz) Primary domain is front-end and
graphic design (also designed our logo).
>> Tomas Veskrna -- starting
>> Patrik Cigas -- starting
>>
>> At this moment we are very close to getting real data to be monitored
via web UI. It’s a matter of 1-2 months considering there is an examination
period happening now at the University.
>>
>> *******************
>> What is InfiSpector and what we want to achieve:
>>
>> * We missed graphical representation of Infinispan nodes communication
so we want
>> -- To be able to spot possible issues AT THE FIRST LOOK (e.g. wait,
this should be coordinator, how is that possible he sends/receives only 10
% of all messages?)
>> -- To demonstrate nicely what’s happening inside of ISPN cluster for
newcomers (to see how Infinispan nodes talk to each other to better
understand concepts)
>> -- To be using nice communication diagrams that describes flows like
(130 messages from node1 to node5 -- click to see them in detail, filter
out in more detail)
>> * We aimed for NON-invasive solution
>> -- No changes in Infinispan internal code
>> -- Just add custom JGroups protocol, gather data and send them where
you want [0]
>> * Provide historical recording of JGroups communication
>> * Help to analyze communication recording from big data point of view
>> -- No need to manually go through gigabytes of text trace logs
>>
>> Simplified InfiSpector architecture:
>>
>> Infinispan Cluster (JGroups with our protocol) ---> Apache Kafka --->
Druid Database (using Kafka Firehose to inject Kafka Topic) <---> NodeJS
server back-end <---> front-end (AngularJS)
>>
>> What is coming out from custom JGroup protocol is a short JSON document
[1] with a timestamp, sending and receiving node, length of a message and
the message itself. Other stuff can be added easily.
>>
>> We will be able to easily answer queries like:
>> How many messages were sent from node1 to node3 during “last” 60
seconds?
>> What are these messages?
>> How many of them were PutKeyValueCommands?
>> Filter out Heart beats (or even ignore them completely), etc.
>>
>> We don’t have any video recording yet but we are very close to that
point. From UI perspective we will be using these 2 charts: [2], [3].
>>
>>
>> Talking about Infinispan 9 plans -- [4] was reported a month ago by you
Galder and we are working on InfiSpector actively let’s say 5 months -- it
looks like I should have advertised InfiSpector more, sooner, but I was
waiting for at least first working demo to start with blogging and videos
:) It’s good that you’ve noticed and that we are having this conversation
right now.
>>
>> To be honest I find
http://zipkin.io/ initiative to be quite similar.
However, InfiSpector seems to be much more “easier” and not targeting at
performance analysis directly. Just adding one protocol at protocol stack
and you are good to go. We were thinking about putting Kafka and Druid
somewhere into the cloud (later) so users don’t need to start all that big
infrastructure at their local machines.
>>
>> I am very open to anything that will help us as a community to achieve
our common goal -- to be able to graphically monitor Infinispan
communication.
>> Additionally I would be _personally_ looking for something that is
easily achievable and is suitable for students to quickly learn new things
and quickly make meaningful contributions.
>>
>> Thanks!
>> Tomas
>>
>> [0] Achieved by custom JGroups protocol -- JGROUPS_TO_KAFKA protocol
has been implemented. This can be added at the end of JGroups stack and
every single message that goes through that is sent to Apache Kafka.
>> [1]
>> {
>> "direction": "receiving/up",
>> "src": "tsykora-19569",
>> "dest": "tsykora-27916",
>> "length": 182,
>> "timestamp": 1460302055376,
>> "message":
"SingleRpcCommand{cacheName='___defaultcache',
command=PutKeyValueCommand{key=f6d52117-8a27-475e-86a7-002a54324615,
value=tsykora-19569, flags=null, putIfAbsent=false,
valueMatcher=MATCH_ALWAYS,
metadata=EmbeddedExpirableMetadata{lifespan=60000, maxIdle=-1,
version=null}, successful=true}}"
>> }
>> [2]
http://bl.ocks.org/NPashaP/9796212
>> [3]
http://bl.ocks.org/mbostock/1046712
>> [4]
https://issues.jboss.org/browse/ISPN-6346
>>
>>
>>
>>
>> ----- Original Message -----
>>> From: "Galder Zamarreño" <galder(a)redhat.com>
>>> To: "infinispan -Dev List" <infinispan-dev(a)lists.jboss.org>,
"Tomas
Sykora" <tsykora(a)redhat.com>
>>> Sent: Monday, May 9, 2016 5:06:06 PM
>>> Subject: Infispector
>>>
>>> Hi all,
>>>
>>> I've just noticed [1], @Thomas, it appears this is your baby? Could you
>>> explain in more detail what you are trying to achieve with this? Do
you have
>>> a video to show what exactly it does?
>>>
>>> Also, who's [2]? Curious to know who's working on this stuff :)
>>>
>>> The reason I'm interested in finding out a bit more about [1] is
because we
>>> have several efforts in the distributed monitoring/tracing area and
want to
>>> make sure we're not duplicating same effort.
>>>
>>> 1. Radim's Message Flow Tracer [3]: This is a project to tool for
tracing
>>> messages and control flow in JGroups/Infinispan using Byteman.
>>>
>>> 2. Zipkin effort [4]: The idea behind is to have a way to have
Infinispan
>>> cluster-wide tracing that uses Zipkin to capture and visualize where
time is
>>> spent within Infinispan. This includes both JGroups and other
components
>>> that could be time consuming, e.g. persistence. This will be main task
for
>>> Infinispan 9. This effort will use a lot of interception points Radim
has
>>> developed in [3] to tie together messages related to a request/tx
around the
>>> cluster.
>>>
>>> Does your effort fall within any of these spaces?
>>>
>>> Cheers,
>>>
>>> [1]
https://github.com/infinispan/infispector
>>> [2]
https://github.com/mciz
>>> [3]
https://github.com/rvansa/message-flow-tracer
>>> [4]
https://issues.jboss.org/browse/ISPN-6346
>>> --
>>> Galder Zamarreño
>>> Infinispan, Red Hat
>>>
>>>
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev(a)lists.jboss.org
>>
https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
>
> --
> Radim Vansa <rvansa(a)redhat.com>
> JBoss Performance Team
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev(a)lists.jboss.org
>
https://lists.jboss.org/mailman/listinfo/infinispan-dev
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev