[infinispan-dev] Infispector

Galder Zamarreño galder at redhat.com
Mon Jun 6 09:39:45 EDT 2016

Hey guys,

Sorry for the delay. I somehow read the replies but forgot to reply :)

I think the target of Infispector and Zipkin based tracing would be different, although there is some common ground. I think both Infispector and Zipkin would be helpful in helping diagnose cluster performance issues and get a general overview of how messages pass from one node to the other. In terms of difference Infispector seems more geared towards education/qe whereas Zipkin is more targeted as something we can run at runtime in production for our users.

Although both projects have different targets I think we'll be able to take advantage of each others work, e.g. custom JGroups protocols points, byteman/bytebuddy rules for intercept scenarions...etc.

Great work leading all the Infispector work Tomas, looking forward to demos/videos :D

Galder Zamarreño
Infinispan, Red Hat

> On 10 May 2016, at 15:53, Radim Vansa <rvansa at redhat.com> wrote:
> To complement this, MFT is a tool that won't offer any sleek charts or 
> visualisations. It's tricky to use and understand - it's intended for 
> developers as a tool for problem analysis. But it gets more in depth 
> than InfiSpector, linking the information from different nodes, JFR 
> events and so forth.
> R.
> On 05/10/2016 09:41 AM, Tomas Sykora wrote:
>> Hello Galder, and all!
>> It’s nice to communicate again via infinispan-dev after a while :)
>> TL;DR: I can see some intersections with zipkin.io initiative goals but InfiSpector seems to be much more “easier to handle and contribute to at this moment” -- that suits more our student-related use case. Let’s continue with the discussion :)
>> Firstly, a short introduction into the context. Red Hat is running Research & Development laboratory here in Brno at 2 biggest local universities: Masaryk University, Faculty of Informatics (FI MU) and Brno University of Technology, Faculty of Information Technologies (FIT VUT).
>> The aim is to better and sooner reach out to students, get them involved into interesting project, show them open-source, git, git workflows and many other things (project specific). An year ago I got excited about this idea and started to think whether I can deliver such a project. And I did.
>> Team faces one big challenge and this is a time constraint. Students are working on _several_ projects during their studies to fulfill courses’ requirements to pass the semester. It’s hard for them to find additional time to be coding even something else. Team managed that but slowly, that’s understandable though. Designing InfiSpector infrastructure took us some time (Kafka, Druid, NodeJS) + evaluation of these technologies + proof of concepts.
>> All 5 team members are 2nd year students of bachelor studies at FIT VUT Brno.
>> Marek Ciz (https://github.com/mciz), also my very good friend from my home town :) His primary domain is Druid, Kafka and infrastructure.
>> Vratislav Hais (https://github.com/vratislavhais) Primary domain is front-end.
>> Jan Fitz (https://github.com/janfitz) Primary domain is front-end and graphic design (also designed our logo).
>> Tomas Veskrna -- starting
>> Patrik Cigas -- starting
>> At this moment we are very close to getting real data to be monitored via web UI. It’s a matter of 1-2 months considering there is an examination period happening now at the University.
>> *******************
>> What is InfiSpector and what we want to achieve:
>> * We missed graphical representation of Infinispan nodes communication so we want
>>   -- To be able to spot possible issues AT THE FIRST LOOK (e.g. wait, this should be coordinator, how is that possible he sends/receives only 10 % of all messages?)
>>   -- To demonstrate nicely what’s happening inside of ISPN cluster for newcomers (to see how Infinispan nodes talk to each other to better understand concepts)
>>   -- To be using nice communication diagrams that describes flows like (130 messages from node1 to node5 -- click to see them in detail, filter out in more detail)
>> * We aimed for NON-invasive solution
>>   -- No changes in Infinispan internal code
>>   -- Just add custom JGroups protocol, gather data and send them where you want [0]
>> * Provide historical recording of JGroups communication
>> * Help to analyze communication recording from big data point of view
>>   -- No need to manually go through gigabytes of text trace logs
>> Simplified InfiSpector architecture:
>> Infinispan Cluster (JGroups with our protocol) ---> Apache Kafka ---> Druid Database (using Kafka Firehose to inject Kafka Topic) <---> NodeJS server back-end <---> front-end (AngularJS)
>> What is coming out from custom JGroup protocol is a short JSON document [1] with a timestamp, sending and receiving node, length of a message and the message itself. Other stuff can be added easily.
>> We will be able to easily answer queries like:
>> How many messages were sent from node1 to node3 during “last” 60 seconds?
>> What are these messages?
>> How many of them were PutKeyValueCommands?
>> Filter out Heart beats (or even ignore them completely), etc.
>> We don’t have any video recording yet but we are very close to that point. From UI perspective we will be using these 2 charts: [2], [3].
>> Talking about Infinispan 9 plans -- [4] was reported a month ago by you Galder and we are working on InfiSpector actively let’s say 5 months -- it looks like I should have advertised InfiSpector more, sooner, but I was waiting for at least first working demo to start with blogging and videos :) It’s good that you’ve noticed and that we are having this conversation right now.
>> To be honest I find http://zipkin.io/ initiative to be quite similar. However, InfiSpector seems to be much more “easier” and not targeting at performance analysis directly. Just adding one protocol at protocol stack and you are good to go. We were thinking about putting Kafka and Druid somewhere into the cloud (later) so users don’t need to start all that big infrastructure at their local machines.
>> I am very open to anything that will help us as a community to achieve our common goal -- to be able to graphically monitor Infinispan communication.
>> Additionally I would be _personally_ looking for something that is easily achievable and is suitable for students to quickly learn new things and quickly make meaningful contributions.
>> Thanks!
>> Tomas
>> [0] Achieved by custom JGroups protocol -- JGROUPS_TO_KAFKA protocol has been implemented. This can be added at the end of JGroups stack and every single message that goes through that is sent to Apache Kafka.
>> [1]
>> {
>>   "direction": "receiving/up",
>>   "src": "tsykora-19569",
>>   "dest": "tsykora-27916",
>>   "length": 182,
>>   "timestamp": 1460302055376,
>>   "message": "SingleRpcCommand{cacheName='___defaultcache', command=PutKeyValueCommand{key=f6d52117-8a27-475e-86a7-002a54324615, value=tsykora-19569, flags=null, putIfAbsent=false, valueMatcher=MATCH_ALWAYS, metadata=EmbeddedExpirableMetadata{lifespan=60000, maxIdle=-1, version=null}, successful=true}}"
>> }
>> [2] http://bl.ocks.org/NPashaP/9796212
>> [3] http://bl.ocks.org/mbostock/1046712
>> [4] https://issues.jboss.org/browse/ISPN-6346
>> ----- Original Message -----
>>> From: "Galder Zamarreño" <galder at redhat.com>
>>> To: "infinispan -Dev List" <infinispan-dev at lists.jboss.org>, "Tomas Sykora" <tsykora at redhat.com>
>>> Sent: Monday, May 9, 2016 5:06:06 PM
>>> Subject: Infispector
>>> Hi all,
>>> I've just noticed [1], @Thomas, it appears this is your baby? Could you
>>> explain in more detail what you are trying to achieve with this? Do you have
>>> a video to show what exactly it does?
>>> Also, who's [2]? Curious to know who's working on this stuff :)
>>> The reason I'm interested in finding out a bit more about [1] is because we
>>> have several efforts in the distributed monitoring/tracing area and want to
>>> make sure we're not duplicating same effort.
>>> 1. Radim's Message Flow Tracer [3]: This is a project to tool for tracing
>>> messages and control flow in JGroups/Infinispan using Byteman.
>>> 2. Zipkin effort [4]: The idea behind is to have a way to have Infinispan
>>> cluster-wide tracing that uses Zipkin to capture and visualize where time is
>>> spent within Infinispan. This includes both JGroups and other components
>>> that could be time consuming, e.g. persistence. This will be main task for
>>> Infinispan 9. This effort will use a lot of interception points Radim has
>>> developed in [3] to tie together messages related to a request/tx around the
>>> cluster.
>>> Does your effort fall within any of these spaces?
>>> Cheers,
>>> [1] https://github.com/infinispan/infispector
>>> [2] https://github.com/mciz
>>> [3] https://github.com/rvansa/message-flow-tracer
>>> [4] https://issues.jboss.org/browse/ISPN-6346
>>> --
>>> Galder Zamarreño
>>> Infinispan, Red Hat
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> -- 
> Radim Vansa <rvansa at redhat.com>
> JBoss Performance Team
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev

More information about the infinispan-dev mailing list