[JBoss JIRA] (ISPN-6833) Visualization and tracing of messages between nodes
by Sebastian Łaskawiec (JIRA)
Sebastian Łaskawiec created ISPN-6833:
-----------------------------------------
Summary: Visualization and tracing of messages between nodes
Key: ISPN-6833
URL: https://issues.jboss.org/browse/ISPN-6833
Project: Infinispan
Issue Type: Feature Request
Reporter: Sebastian Łaskawiec
Note: There is a work in progress already happening: https://github.com/infinispan/infispector
We know what kind of messages should be generated between nodes to perform specific operations - in theory -, still to debug problems of configuration or implementation on the whole stack (application + Infinispan + JGroups) we often need to look into the logs, having thousands of trace lines even when sampling for small periods of time.
It would be very useful to have a way to automatically extrapolate the interesting patterns out of a running system, we could collect reliable information for example using (just ideas):
* A custom JGroups protocol
* Byteman to instrument JGroups for specific events (like network socket usage, or thresholds being reached in internal structures like resend tables or threadpool sizes)
* Simple log file parsing
The collected information could then be used to generate condensed reports highlighting the patterns being used in practice to compare them with expected patterns.
I have two different kinds if output in mind:
* A graphical visualization, showing the cluster nodes and a sequence of colored arrows showcasing what is being done
* A short text representation, to be used by:
** automated tests to verify invariant expectations are not broken on code changes
** future possible tool to formally proof correctness / race conditions
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
7 years, 9 months
[JBoss JIRA] (ISPN-6832) Proof of correctness for complex distributed patterns
by Sebastian Łaskawiec (JIRA)
Sebastian Łaskawiec created ISPN-6832:
-----------------------------------------
Summary: Proof of correctness for complex distributed patterns
Key: ISPN-6832
URL: https://issues.jboss.org/browse/ISPN-6832
Project: Infinispan
Issue Type: Feature Request
Reporter: Sebastian Łaskawiec
The core of Infinispan can be represented in very simple "primitives": a set of nodes send messages to each other. The fundamental rules are also relatively simple:
* a message can't be received before it's sent
* a message could be lost
* a node could be killed at any time
>From these basic building blocks one can start building some logical consequences:
* messages sent to multiple nodes might arrive at different times
* multiple messages sent to multiple nodes might be delivered in different order
Some of these problems are resolved by JGroups - but even then it must be configured accordingly, meaning Infinispan' s usage of it is sensible to using the wrong method or flags.
While the Java/Scala code in Infinispan is not overly complex, it's not suited for reasoning on the consequences of often-needed changes in the codebase. It would be very useful to be able to define patterns in an ad-hoc meta language, and provide a proof correctness of the patterns it uses, or at least proof the events it should avoid can not happen.
A great help for the project would be to sketch such a language and try it out on some of the distribution schemes Infinispan uses to proof they are correct or identify flaws in correctness. In a second step (optional) one could build some tooling around this to provide automatic demonstration / simulation for proposed changes expressed in this meta language.
The Promela language could be used for this purpose; so one could build tooling around it, try to apply it on Infinispan, and possibly work on ad-hoc extesions.
See also previous proposal: "Visualization and tracing of messages between nodes" -> that would make it possible to trace the real Infinispan behaviour and then demonstrate it's correctness or identify problems before they happen.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
7 years, 9 months
[JBoss JIRA] (ISPN-6831) Implement topology changes updates in Ruby client
by Sebastian Łaskawiec (JIRA)
Sebastian Łaskawiec created ISPN-6831:
-----------------------------------------
Summary: Implement topology changes updates in Ruby client
Key: ISPN-6831
URL: https://issues.jboss.org/browse/ISPN-6831
Project: Infinispan
Issue Type: Feature Request
Reporter: Sebastian Łaskawiec
Hot Rod protocol is a binary, platform independent, protocol created to enable clients to communicate with Infinispan servers. Hot Rod protocol clients can receive, as part of operation responses, cluster topology update information. The aim of this task is to implement this logic in Hot Rod's Ruby client.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
7 years, 9 months