[infinispan-dev] Musings on ISPN/JGRPs OSI transport choices and ambitions

[Bela Ban]

Hi Ben,

why don't you post an edited version of my private replies to you to 
this topic as well, so we have some background ?

In a nutshell, you want to use a zero-copy transport between processes 
that run on the *same physical box* (e.g. shipping only pointers to 
native shared memory between processes). Currently, using TCP or UDP 
between processes on the same box still requires 1 or 2 copies, even 
when a loopback device is used.

I'm interested in adding such a transport in JGroups 4, in which I plan 
to revamp the transport to adopt an NIO based scheme, accommodating both 
TDP and TCP. This is all still in the planning phase, but one feature 
will be to have multiple transports running in the same stack and 
sending messages alternatively via different transports. E.g. multicasts 
would use UDP whereas unicasts would use TCP (by default), but this 
could be overridden per message (with flags).

If we then had 5 physical boxes, with 20 processes on each box, for a 
total of 100 nodes, then we could configure the stacks to run both 
SHR_MEM and UDP: a group-wide message (a multicast) would be sent via 
SHR_MEM *and* UDP.

The SHR_MEM transport would disseminate the message to all 20 processes 
on the same physical box, using shared memory. The UDP transport would 
be configured as non-loopback (IP_MULTICAST_LOOP=false), which means 
that the message would be multicast to the other 3 physical boxes, but 
the local multicast would be dropped. The other boxes would then use 
SHR_MEM to disseminate the message locally to all 20 processes.

Just an idea atm, this could also be done via RELAY2, but the QoS would 
not be the same.

I'm planning on releasing 3.5 in 6-8 weeks from now. This includes a 
community baking phase during which I'll be working on a deep-dive 
course on JGroups.

So a *very tentative* schedule is to start on 4.0 at the beginning of 
summer.


On 28/02/14 19:16, cotton-ben wrote:
> Hi Mircea, Manik, Bela, et. al.
>
> I want to more publicly muse on this SUBJ line.  Here now, then maybe in
> ISPN /user/ forum, then maybe JSR-347 provider wide.  I know we had a
> semi-private (Bela led) exchange, but I want to be more public with this
> conversation.
>
> Long post again.  sorry.
>
> This is just on open musing.  I realize this musing should not expect to be
> accommodated by any "oh, we got to do this in ISPN/JGRPs now!" repsonse ...
> there is absolutely only the most infrequent use-case that would /today/ be
> served by addressing this musing ... but tomorrow that /will/ be a different
> story.
>
> Questions::
>
> Does the concept of ISPN/JGRPs  transport between "Cluster" nodes currently
> depend on OSI transport layer sockets' participation(s)?
>
> In other words, if all the nodes on my "Cluster" have locality=127.0.0.1  is
> ISPN/JGRPs  accommodating  enough to use a native OS IPC choice as an
> intra-node transport?
>
> Or, is it true that my transport choices are always limited to just
> {TCP,UDP} --  independent of the participating nodes' locality (and that I
> am thus forced to go over an OSI loopback)?
>
> If my transport choices are only limited to {TCP,UDP} for all node locality,
> then I might ask that you consider additional upcoming modern Java transport
> options.
>
>   With the ambitions of upcoming OpenJDK JEPs,  that will make mainstream an
> API capabilty that today is only available via sun.misc.Unsafe, Java will
> soon have "more complete" transport options that will include all of
>
>   { TCP, UDP,  RDMA/SDP,   IPC }
>
> Some examples of upcoming accommodating providers=
>
> 1.  RDMA/SDP: via  Infiniband VERBS (works today in JDK 7 on OSI physical
> layer IB NICs, does not work over Ethernet)
> 2.  IPC via OpenHFT' SHM as IPC solution (will work this year)
>
> Again, I realize that these transport choices are useful today only  in a
> very rare use case.  However, should these transports be in your offering to
> ISPN/JGRPs customers, then ISPN/JGRPs becomes   -- like all of Java has
> become in recent years --  increasingly more attractive to /all/ HPC Linux
> supercomputing use cases (not just ours).
>
>
>
>
>
>
>
> --
> View this message in context: http://infinispan-developer-list.980875.n3.nabble.com/Musings-on-ISPN-JGRPs-OSI-transport-choices-and-ambitions-tp4028925.html
> Sent from the Infinispan Developer List mailing list archive at Nabble.com.
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>

-- 
Bela Ban, JGroups lead (http://www.jgroups.org)