Re: [infinispan-dev] Infispector

Hello Galder, and all! It’s nice to communicate again via infinispan-dev after a while :) TL;DR: I can see some intersections with zipkin.io initiative goals but InfiSpector seems to be much more “easier to handle and contribute to at this moment” -- that suits more our student-related use case. Let’s continue with the discussion :) Firstly, a short introduction into the context. Red Hat is running Research & Development laboratory here in Brno at 2 biggest local universities: Masaryk University, Faculty of Informatics (FI MU) and Brno University of Technology, Faculty of Information Technologies (FIT VUT). The aim is to better and sooner reach out to students, get them involved into interesting project, show them open-source, git, git workflows and many other things (project specific). An year ago I got excited about this idea and started to think whether I can deliver such a project. And I did. Team faces one big challenge and this is a time constraint. Students are working on _several_ projects during their studies to fulfill courses’ requirements to pass the semester. It’s hard for them to find additional time to be coding even something else. Team managed that but slowly, that’s understandable though. Designing InfiSpector infrastructure took us some time (Kafka, Druid, NodeJS) + evaluation of these technologies + proof of concepts. All 5 team members are 2nd year students of bachelor studies at FIT VUT Brno. Marek Ciz (https://github.com/mciz), also my very good friend from my home town :) His primary domain is Druid, Kafka and infrastructure. Vratislav Hais (https://github.com/vratislavhais) Primary domain is front-end. Jan Fitz (https://github.com/janfitz) Primary domain is front-end and graphic design (also designed our logo). Tomas Veskrna -- starting Patrik Cigas -- starting At this moment we are very close to getting real data to be monitored via web UI. It’s a matter of 1-2 months considering there is an examination period happening now at the University. ******************* What is InfiSpector and what we want to achieve: * We missed graphical representation of Infinispan nodes communication so we want -- To be able to spot possible issues AT THE FIRST LOOK (e.g. wait, this should be coordinator, how is that possible he sends/receives only 10 % of all messages?) -- To demonstrate nicely what’s happening inside of ISPN cluster for newcomers (to see how Infinispan nodes talk to each other to better understand concepts) -- To be using nice communication diagrams that describes flows like (130 messages from node1 to node5 -- click to see them in detail, filter out in more detail) * We aimed for NON-invasive solution -- No changes in Infinispan internal code -- Just add custom JGroups protocol, gather data and send them where you want [0] * Provide historical recording of JGroups communication * Help to analyze communication recording from big data point of view -- No need to manually go through gigabytes of text trace logs Simplified InfiSpector architecture: Infinispan Cluster (JGroups with our protocol) ---> Apache Kafka ---> Druid Database (using Kafka Firehose to inject Kafka Topic) <---> NodeJS server back-end <---> front-end (AngularJS) What is coming out from custom JGroup protocol is a short JSON document [1] with a timestamp, sending and receiving node, length of a message and the message itself. Other stuff can be added easily. We will be able to easily answer queries like: How many messages were sent from node1 to node3 during “last” 60 seconds? What are these messages? How many of them were PutKeyValueCommands? Filter out Heart beats (or even ignore them completely), etc. We don’t have any video recording yet but we are very close to that point. From UI perspective we will be using these 2 charts: [2], [3]. Talking about Infinispan 9 plans -- [4] was reported a month ago by you Galder and we are working on InfiSpector actively let’s say 5 months -- it looks like I should have advertised InfiSpector more, sooner, but I was waiting for at least first working demo to start with blogging and videos :) It’s good that you’ve noticed and that we are having this conversation right now. To be honest I find http://zipkin.io/ initiative to be quite similar. However, InfiSpector seems to be much more “easier” and not targeting at performance analysis directly. Just adding one protocol at protocol stack and you are good to go. We were thinking about putting Kafka and Druid somewhere into the cloud (later) so users don’t need to start all that big infrastructure at their local machines. I am very open to anything that will help us as a community to achieve our common goal -- to be able to graphically monitor Infinispan communication. Additionally I would be _personally_ looking for something that is easily achievable and is suitable for students to quickly learn new things and quickly make meaningful contributions. Thanks! Tomas [0] Achieved by custom JGroups protocol -- JGROUPS_TO_KAFKA protocol has been implemented. This can be added at the end of JGroups stack and every single message that goes through that is sent to Apache Kafka. [1] { "direction": "receiving/up", "src": "tsykora-19569", "dest": "tsykora-27916", "length": 182, "timestamp": 1460302055376, "message": "SingleRpcCommand{cacheName='___defaultcache', command=PutKeyValueCommand{key=f6d52117-8a27-475e-86a7-002a54324615, value=tsykora-19569, flags=null, putIfAbsent=false, valueMatcher=MATCH_ALWAYS, metadata=EmbeddedExpirableMetadata{lifespan=60000, maxIdle=-1, version=null}, successful=true}}" } [2] http://bl.ocks.org/NPashaP/9796212 [3] http://bl.ocks.org/mbostock/1046712 [4] https://issues.jboss.org/browse/ISPN-6346 ----- Original Message -----