[infinispan-dev] Infispector

Sebastian Laskawiec slaskawi at redhat.com
Wed Jun 8 03:22:37 EDT 2016


+1000 looks very interesting!

I think we should a nice visualization app for our conference
presentations... We might consider refreshing Visual [5], but personally I
would love to see something new. A NodeJS client might be also a nice way
to go but tracing how the data flows through the cluster... this would be
something nice...

Thanks
Sebastian

[5] https://github.com/infinispan/visual

On Mon, Jun 6, 2016 at 3:39 PM, Galder Zamarreño <galder at redhat.com> wrote:

> Hey guys,
>
> Sorry for the delay. I somehow read the replies but forgot to reply :)
>
> I think the target of Infispector and Zipkin based tracing would be
> different, although there is some common ground. I think both Infispector
> and Zipkin would be helpful in helping diagnose cluster performance issues
> and get a general overview of how messages pass from one node to the other.
> In terms of difference Infispector seems more geared towards education/qe
> whereas Zipkin is more targeted as something we can run at runtime in
> production for our users.
>
> Although both projects have different targets I think we'll be able to
> take advantage of each others work, e.g. custom JGroups protocols points,
> byteman/bytebuddy rules for intercept scenarions...etc.
>
> Great work leading all the Infispector work Tomas, looking forward to
> demos/videos :D
>
> Cheers,
> --
> Galder Zamarreño
> Infinispan, Red Hat
>
> > On 10 May 2016, at 15:53, Radim Vansa <rvansa at redhat.com> wrote:
> >
> > To complement this, MFT is a tool that won't offer any sleek charts or
> > visualisations. It's tricky to use and understand - it's intended for
> > developers as a tool for problem analysis. But it gets more in depth
> > than InfiSpector, linking the information from different nodes, JFR
> > events and so forth.
> >
> > R.
> >
> > On 05/10/2016 09:41 AM, Tomas Sykora wrote:
> >> Hello Galder, and all!
> >> It’s nice to communicate again via infinispan-dev after a while :)
> >>
> >> TL;DR: I can see some intersections with zipkin.io initiative goals
> but InfiSpector seems to be much more “easier to handle and contribute to
> at this moment” -- that suits more our student-related use case. Let’s
> continue with the discussion :)
> >>
> >> Firstly, a short introduction into the context. Red Hat is running
> Research & Development laboratory here in Brno at 2 biggest local
> universities: Masaryk University, Faculty of Informatics (FI MU) and Brno
> University of Technology, Faculty of Information Technologies (FIT VUT).
> >> The aim is to better and sooner reach out to students, get them
> involved into interesting project, show them open-source, git, git
> workflows and many other things (project specific). An year ago I got
> excited about this idea and started to think whether I can deliver such a
> project. And I did.
> >>
> >> Team faces one big challenge and this is a time constraint. Students
> are working on _several_ projects during their studies to fulfill courses’
> requirements to pass the semester. It’s hard for them to find additional
> time to be coding even something else. Team managed that but slowly, that’s
> understandable though. Designing InfiSpector infrastructure took us some
> time (Kafka, Druid, NodeJS) + evaluation of these technologies + proof of
> concepts.
> >>
> >> All 5 team members are 2nd year students of bachelor studies at FIT VUT
> Brno.
> >> Marek Ciz (https://github.com/mciz), also my very good friend from my
> home town :) His primary domain is Druid, Kafka and infrastructure.
> >> Vratislav Hais (https://github.com/vratislavhais) Primary domain is
> front-end.
> >> Jan Fitz (https://github.com/janfitz) Primary domain is front-end and
> graphic design (also designed our logo).
> >> Tomas Veskrna -- starting
> >> Patrik Cigas -- starting
> >>
> >> At this moment we are very close to getting real data to be monitored
> via web UI. It’s a matter of 1-2 months considering there is an examination
> period happening now at the University.
> >>
> >> *******************
> >> What is InfiSpector and what we want to achieve:
> >>
> >> * We missed graphical representation of Infinispan nodes communication
> so we want
> >>   -- To be able to spot possible issues AT THE FIRST LOOK (e.g. wait,
> this should be coordinator, how is that possible he sends/receives only 10
> % of all messages?)
> >>   -- To demonstrate nicely what’s happening inside of ISPN cluster for
> newcomers (to see how Infinispan nodes talk to each other to better
> understand concepts)
> >>   -- To be using nice communication diagrams that describes flows like
> (130 messages from node1 to node5 -- click to see them in detail, filter
> out in more detail)
> >> * We aimed for NON-invasive solution
> >>   -- No changes in Infinispan internal code
> >>   -- Just add custom JGroups protocol, gather data and send them where
> you want [0]
> >> * Provide historical recording of JGroups communication
> >> * Help to analyze communication recording from big data point of view
> >>   -- No need to manually go through gigabytes of text trace logs
> >>
> >> Simplified InfiSpector architecture:
> >>
> >> Infinispan Cluster (JGroups with our protocol) ---> Apache Kafka --->
> Druid Database (using Kafka Firehose to inject Kafka Topic) <---> NodeJS
> server back-end <---> front-end (AngularJS)
> >>
> >> What is coming out from custom JGroup protocol is a short JSON document
> [1] with a timestamp, sending and receiving node, length of a message and
> the message itself. Other stuff can be added easily.
> >>
> >> We will be able to easily answer queries like:
> >> How many messages were sent from node1 to node3 during “last” 60
> seconds?
> >> What are these messages?
> >> How many of them were PutKeyValueCommands?
> >> Filter out Heart beats (or even ignore them completely), etc.
> >>
> >> We don’t have any video recording yet but we are very close to that
> point. From UI perspective we will be using these 2 charts: [2], [3].
> >>
> >>
> >> Talking about Infinispan 9 plans -- [4] was reported a month ago by you
> Galder and we are working on InfiSpector actively let’s say 5 months -- it
> looks like I should have advertised InfiSpector more, sooner, but I was
> waiting for at least first working demo to start with blogging and videos
> :) It’s good that you’ve noticed and that we are having this conversation
> right now.
> >>
> >> To be honest I find http://zipkin.io/ initiative to be quite similar.
> However, InfiSpector seems to be much more “easier” and not targeting at
> performance analysis directly. Just adding one protocol at protocol stack
> and you are good to go. We were thinking about putting Kafka and Druid
> somewhere into the cloud (later) so users don’t need to start all that big
> infrastructure at their local machines.
> >>
> >> I am very open to anything that will help us as a community to achieve
> our common goal -- to be able to graphically monitor Infinispan
> communication.
> >> Additionally I would be _personally_ looking for something that is
> easily achievable and is suitable for students to quickly learn new things
> and quickly make meaningful contributions.
> >>
> >> Thanks!
> >> Tomas
> >>
> >> [0] Achieved by custom JGroups protocol -- JGROUPS_TO_KAFKA protocol
> has been implemented. This can be added at the end of JGroups stack and
> every single message that goes through that is sent to Apache Kafka.
> >> [1]
> >> {
> >>   "direction": "receiving/up",
> >>   "src": "tsykora-19569",
> >>   "dest": "tsykora-27916",
> >>   "length": 182,
> >>   "timestamp": 1460302055376,
> >>   "message": "SingleRpcCommand{cacheName='___defaultcache',
> command=PutKeyValueCommand{key=f6d52117-8a27-475e-86a7-002a54324615,
> value=tsykora-19569, flags=null, putIfAbsent=false,
> valueMatcher=MATCH_ALWAYS,
> metadata=EmbeddedExpirableMetadata{lifespan=60000, maxIdle=-1,
> version=null}, successful=true}}"
> >> }
> >> [2] http://bl.ocks.org/NPashaP/9796212
> >> [3] http://bl.ocks.org/mbostock/1046712
> >> [4] https://issues.jboss.org/browse/ISPN-6346
> >>
> >>
> >>
> >>
> >> ----- Original Message -----
> >>> From: "Galder Zamarreño" <galder at redhat.com>
> >>> To: "infinispan -Dev List" <infinispan-dev at lists.jboss.org>, "Tomas
> Sykora" <tsykora at redhat.com>
> >>> Sent: Monday, May 9, 2016 5:06:06 PM
> >>> Subject: Infispector
> >>>
> >>> Hi all,
> >>>
> >>> I've just noticed [1], @Thomas, it appears this is your baby? Could you
> >>> explain in more detail what you are trying to achieve with this? Do
> you have
> >>> a video to show what exactly it does?
> >>>
> >>> Also, who's [2]? Curious to know who's working on this stuff :)
> >>>
> >>> The reason I'm interested in finding out a bit more about [1] is
> because we
> >>> have several efforts in the distributed monitoring/tracing area and
> want to
> >>> make sure we're not duplicating same effort.
> >>>
> >>> 1. Radim's Message Flow Tracer [3]: This is a project to tool for
> tracing
> >>> messages and control flow in JGroups/Infinispan using Byteman.
> >>>
> >>> 2. Zipkin effort [4]: The idea behind is to have a way to have
> Infinispan
> >>> cluster-wide tracing that uses Zipkin to capture and visualize where
> time is
> >>> spent within Infinispan. This includes both JGroups and other
> components
> >>> that could be time consuming, e.g. persistence. This will be main task
> for
> >>> Infinispan 9. This effort will use a lot of interception points Radim
> has
> >>> developed in [3] to tie together messages related to a request/tx
> around the
> >>> cluster.
> >>>
> >>> Does your effort fall within any of these spaces?
> >>>
> >>> Cheers,
> >>>
> >>> [1] https://github.com/infinispan/infispector
> >>> [2] https://github.com/mciz
> >>> [3] https://github.com/rvansa/message-flow-tracer
> >>> [4] https://issues.jboss.org/browse/ISPN-6346
> >>> --
> >>> Galder Zamarreño
> >>> Infinispan, Red Hat
> >>>
> >>>
> >> _______________________________________________
> >> infinispan-dev mailing list
> >> infinispan-dev at lists.jboss.org
> >> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >
> >
> > --
> > Radim Vansa <rvansa at redhat.com>
> > JBoss Performance Team
> >
> > _______________________________________________
> > infinispan-dev mailing list
> > infinispan-dev at lists.jboss.org
> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/infinispan-dev/attachments/20160608/902a9aee/attachment.html 


More information about the infinispan-dev mailing list