Hey guys,
Sorry for the delay. I somehow read the replies but forgot to reply :)
I think the target of Infispector and Zipkin based tracing would be different, although
there is some common ground. I think both Infispector and Zipkin would be helpful in
helping diagnose cluster performance issues and get a general overview of how messages
pass from one node to the other. In terms of difference Infispector seems more geared
towards education/qe whereas Zipkin is more targeted as something we can run at runtime in
production for our users.
Although both projects have different targets I think we'll be able to take advantage
of each others work, e.g. custom JGroups protocols points, byteman/bytebuddy rules for
intercept scenarions...etc.
Great work leading all the Infispector work Tomas, looking forward to demos/videos :D
Cheers,
--
Galder Zamarreño
Infinispan, Red Hat
On 10 May 2016, at 15:53, Radim Vansa <rvansa(a)redhat.com>
wrote:
To complement this, MFT is a tool that won't offer any sleek charts or
visualisations. It's tricky to use and understand - it's intended for
developers as a tool for problem analysis. But it gets more in depth
than InfiSpector, linking the information from different nodes, JFR
events and so forth.
R.
On 05/10/2016 09:41 AM, Tomas Sykora wrote:
> Hello Galder, and all!
> It’s nice to communicate again via infinispan-dev after a while :)
>
> TL;DR: I can see some intersections with zipkin.io initiative goals but InfiSpector
seems to be much more “easier to handle and contribute to at this moment” -- that suits
more our student-related use case. Let’s continue with the discussion :)
>
> Firstly, a short introduction into the context. Red Hat is running Research &
Development laboratory here in Brno at 2 biggest local universities: Masaryk University,
Faculty of Informatics (FI MU) and Brno University of Technology, Faculty of Information
Technologies (FIT VUT).
> The aim is to better and sooner reach out to students, get them involved into
interesting project, show them open-source, git, git workflows and many other things
(project specific). An year ago I got excited about this idea and started to think whether
I can deliver such a project. And I did.
>
> Team faces one big challenge and this is a time constraint. Students are working on
_several_ projects during their studies to fulfill courses’ requirements to pass the
semester. It’s hard for them to find additional time to be coding even something else.
Team managed that but slowly, that’s understandable though. Designing InfiSpector
infrastructure took us some time (Kafka, Druid, NodeJS) + evaluation of these technologies
+ proof of concepts.
>
> All 5 team members are 2nd year students of bachelor studies at FIT VUT Brno.
> Marek Ciz (
https://github.com/mciz), also my very good friend from my home town :)
His primary domain is Druid, Kafka and infrastructure.
> Vratislav Hais (
https://github.com/vratislavhais) Primary domain is front-end.
> Jan Fitz (
https://github.com/janfitz) Primary domain is front-end and graphic design
(also designed our logo).
> Tomas Veskrna -- starting
> Patrik Cigas -- starting
>
> At this moment we are very close to getting real data to be monitored via web UI.
It’s a matter of 1-2 months considering there is an examination period happening now at
the University.
>
> *******************
> What is InfiSpector and what we want to achieve:
>
> * We missed graphical representation of Infinispan nodes communication so we want
> -- To be able to spot possible issues AT THE FIRST LOOK (e.g. wait, this should be
coordinator, how is that possible he sends/receives only 10 % of all messages?)
> -- To demonstrate nicely what’s happening inside of ISPN cluster for newcomers (to
see how Infinispan nodes talk to each other to better understand concepts)
> -- To be using nice communication diagrams that describes flows like (130 messages
from node1 to node5 -- click to see them in detail, filter out in more detail)
> * We aimed for NON-invasive solution
> -- No changes in Infinispan internal code
> -- Just add custom JGroups protocol, gather data and send them where you want [0]
> * Provide historical recording of JGroups communication
> * Help to analyze communication recording from big data point of view
> -- No need to manually go through gigabytes of text trace logs
>
> Simplified InfiSpector architecture:
>
> Infinispan Cluster (JGroups with our protocol) ---> Apache Kafka ---> Druid
Database (using Kafka Firehose to inject Kafka Topic) <---> NodeJS server back-end
<---> front-end (AngularJS)
>
> What is coming out from custom JGroup protocol is a short JSON document [1] with a
timestamp, sending and receiving node, length of a message and the message itself. Other
stuff can be added easily.
>
> We will be able to easily answer queries like:
> How many messages were sent from node1 to node3 during “last” 60 seconds?
> What are these messages?
> How many of them were PutKeyValueCommands?
> Filter out Heart beats (or even ignore them completely), etc.
>
> We don’t have any video recording yet but we are very close to that point. From UI
perspective we will be using these 2 charts: [2], [3].
>
>
> Talking about Infinispan 9 plans -- [4] was reported a month ago by you Galder and we
are working on InfiSpector actively let’s say 5 months -- it looks like I should have
advertised InfiSpector more, sooner, but I was waiting for at least first working demo to
start with blogging and videos :) It’s good that you’ve noticed and that we are having
this conversation right now.
>
> To be honest I find
http://zipkin.io/ initiative to be quite similar. However,
InfiSpector seems to be much more “easier” and not targeting at performance analysis
directly. Just adding one protocol at protocol stack and you are good to go. We were
thinking about putting Kafka and Druid somewhere into the cloud (later) so users don’t
need to start all that big infrastructure at their local machines.
>
> I am very open to anything that will help us as a community to achieve our common
goal -- to be able to graphically monitor Infinispan communication.
> Additionally I would be _personally_ looking for something that is easily achievable
and is suitable for students to quickly learn new things and quickly make meaningful
contributions.
>
> Thanks!
> Tomas
>
> [0] Achieved by custom JGroups protocol -- JGROUPS_TO_KAFKA protocol has been
implemented. This can be added at the end of JGroups stack and every single message that
goes through that is sent to Apache Kafka.
> [1]
> {
> "direction": "receiving/up",
> "src": "tsykora-19569",
> "dest": "tsykora-27916",
> "length": 182,
> "timestamp": 1460302055376,
> "message": "SingleRpcCommand{cacheName='___defaultcache',
command=PutKeyValueCommand{key=f6d52117-8a27-475e-86a7-002a54324615, value=tsykora-19569,
flags=null, putIfAbsent=false, valueMatcher=MATCH_ALWAYS,
metadata=EmbeddedExpirableMetadata{lifespan=60000, maxIdle=-1, version=null},
successful=true}}"
> }
> [2]
http://bl.ocks.org/NPashaP/9796212
> [3]
http://bl.ocks.org/mbostock/1046712
> [4]
https://issues.jboss.org/browse/ISPN-6346
>
>
>
>
> ----- Original Message -----
>> From: "Galder Zamarreño" <galder(a)redhat.com>
>> To: "infinispan -Dev List" <infinispan-dev(a)lists.jboss.org>,
"Tomas Sykora" <tsykora(a)redhat.com>
>> Sent: Monday, May 9, 2016 5:06:06 PM
>> Subject: Infispector
>>
>> Hi all,
>>
>> I've just noticed [1], @Thomas, it appears this is your baby? Could you
>> explain in more detail what you are trying to achieve with this? Do you have
>> a video to show what exactly it does?
>>
>> Also, who's [2]? Curious to know who's working on this stuff :)
>>
>> The reason I'm interested in finding out a bit more about [1] is because we
>> have several efforts in the distributed monitoring/tracing area and want to
>> make sure we're not duplicating same effort.
>>
>> 1. Radim's Message Flow Tracer [3]: This is a project to tool for tracing
>> messages and control flow in JGroups/Infinispan using Byteman.
>>
>> 2. Zipkin effort [4]: The idea behind is to have a way to have Infinispan
>> cluster-wide tracing that uses Zipkin to capture and visualize where time is
>> spent within Infinispan. This includes both JGroups and other components
>> that could be time consuming, e.g. persistence. This will be main task for
>> Infinispan 9. This effort will use a lot of interception points Radim has
>> developed in [3] to tie together messages related to a request/tx around the
>> cluster.
>>
>> Does your effort fall within any of these spaces?
>>
>> Cheers,
>>
>> [1]
https://github.com/infinispan/infispector
>> [2]
https://github.com/mciz
>> [3]
https://github.com/rvansa/message-flow-tracer
>> [4]
https://issues.jboss.org/browse/ISPN-6346
>> --
>> Galder Zamarreño
>> Infinispan, Red Hat
>>
>>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev(a)lists.jboss.org
>
https://lists.jboss.org/mailman/listinfo/infinispan-dev
--
Radim Vansa <rvansa(a)redhat.com>
JBoss Performance Team
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev