<meta http-equiv=Content-Type content="text/html; charset=iso-8859-1"><meta name=Generator content="Microsoft Word 14 (filtered medium)"><!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--><div class=WordSection1><p class=MsoListParagraph style='text-indent:-.25in;mso-list:l1 level1 lfo1'><span style='font-family:Wingdings;color:#7F7F7F;mso-style-textfill-fill-color:#7F7F7F;mso-style-textfill-fill-alpha:100.0%'><span style='mso-list:Ignore'>Ø<span style='font:7.0pt "Times New Roman"'> </span></span></span>In a nutshell, you want to use a zero-copy transport between processes <br>that run on the *same physical box* (e.g. shipping only pointers to <br>native shared memory between processes).<span style='font-family:"Calibri","sans-serif";color:#1F497D'><o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-family:"Calibri","sans-serif";color:blue'>This is exactly what we want. <o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Calibri","sans-serif";color:blue'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-family:"Calibri","sans-serif";color:blue'>Without this capability our beloved RedHat ISPN/JGRPS stack pays no homage to our architected locality premiums. Remember when EJB made the fatal mistake of forcing remoteness on its API’s end-user? Just plain LAZY these EJB designers! How dare they? They ended up with the grossly inelegant hack of providing a </span><span style='font-family:"Lucida Console";color:blue'>localremote() </span><span style='font-family:"Calibri","sans-serif";color:blue'> interface. OBSCENE. When you hack up something to the point of being forced to cattle-prod the API with something called “</span><span style='font-family:"Lucida Console";color:blue'>localremote()</span><span style='font-family:"Calibri","sans-serif";color:blue'>” (a <i>literal </i>oxymoron in your API. Are you kidding me?) you just have to laugh and say “damn that is BROKE!”.<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Calibri","sans-serif";color:blue'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-family:"Calibri","sans-serif";color:blue'>It prejudiced Java in the eyes of the HPC community. For years.<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Calibri","sans-serif";color:blue'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-family:"Calibri","sans-serif";color:blue'>Now Bela, we totally respect that you won’t ever put C++/JNI into JGRPs – even to accommodate our architected supercomputing locality premiums. We get that. But, now Bela! But now! We have a 100% Java solution to take us where we want to go – an IPC transport over /dev/shm – without a single line of C++ nor JNI. It is beautiful. It solves a <a href="https://groups.google.com/forum/#!topic/mechanical-sympathy/rBG7hcamt1k" target="_top" rel="nofollow" link="external">problem of ours</a>. <o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Calibri","sans-serif";color:blue'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-family:"Calibri","sans-serif";color:blue'>By doing this soon in JGRPs, Bela, you can realize two beautiful outcomes:<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Calibri","sans-serif";color:blue'><o:p> </o:p></span></p><p class=MsoListParagraph style='text-indent:-.25in;mso-list:l0 level1 lfo2'><span style='font-family:"Calibri","sans-serif";color:blue'><span style='mso-list:Ignore'>1.<span style='font:7.0pt "Times New Roman"'> </span></span></span><span style='font-family:"Calibri","sans-serif";color:blue'> JGRPs remains magnifique: 100% Pure Java, not a single JNI bridge to C++ to native kernel system calls (we agree that is unattractive)<o:p></o:p></span></p><p class=MsoListParagraph style='text-indent:-.25in;mso-list:l0 level1 lfo2'><span style='font-family:"Calibri","sans-serif";color:blue'><span style='mso-list:Ignore'>2.<span style='font:7.0pt "Times New Roman"'> </span></span></span><span style='font-family:"Calibri","sans-serif";color:blue'>You liberate the JGRPs end-user from having remoteness (which OSI loopback is) forced down their throats <o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Calibri","sans-serif";color:blue'><o:p> </o:p></span></p><p class=MsoListParagraph style='text-indent:-.25in;mso-list:l1 level1 lfo1'><span style='font-family:Wingdings;color:#7F7F7F;mso-style-textfill-fill-color:#7F7F7F;mso-style-textfill-fill-alpha:100.0%'><span style='mso-list:Ignore'>Ø<span style='font:7.0pt "Times New Roman"'> </span></span></span>I'm interested in adding such a transport in JGroups 4, <span style='font-family:"Calibri","sans-serif";color:#1F497D'><o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-family:"Calibri","sans-serif";color:blue'>This thrills us!<o:p></o:p></span></p><p class=MsoListParagraph><o:p> </o:p></p><p class=MsoListParagraph style='text-indent:-.25in;mso-list:l1 level1 lfo1'><span style='font-family:Wingdings;color:#7F7F7F;mso-style-textfill-fill-color:#7F7F7F;mso-style-textfill-fill-alpha:100.0%'><span style='mso-list:Ignore'>Ø<span style='font:7.0pt "Times New Roman"'> </span></span></span>in which I plan to revamp the transport to adopt an NIO based scheme<span style='font-family:"Calibri","sans-serif";color:#1F497D'><o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-family:"Calibri","sans-serif";color:blue'>I may be mistaken, but I think the OpenHFT solution for using SHM as an IPC transport has big advantages over using the NIO bridges to Off-Heap capabilities. Doesn’t NIO’s DirectByteBuffer have a capacity delinquency that will immediately betray big data views? I’m not sure about this, I’ll get back to you (publicly, here).<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Calibri","sans-serif";color:blue'><o:p> </o:p></span></p><p class=MsoListParagraph style='text-indent:-.25in;mso-list:l1 level1 lfo1'><span style='font-family:Wingdings;color:#7F7F7F;mso-style-textfill-fill-color:#7F7F7F;mso-style-textfill-fill-alpha:100.0%'><span style='mso-list:Ignore'>Ø<span style='font:7.0pt "Times New Roman"'> </span></span></span>a group-wide message (a multicast) would be sent via SHR_MEM *and* UDP. <span style='font-family:"Calibri","sans-serif";color:blue'><o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Calibri","sans-serif";color:blue'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-family:"Calibri","sans-serif";color:blue'>Perfect.<o:p></o:p></span></p><p class=MsoListParagraph><span style='font-family:"Calibri","sans-serif";color:blue'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoListParagraph style='text-indent:-.25in;mso-list:l1 level1 lfo1'><span style='font-family:Wingdings;color:#7F7F7F;mso-style-textfill-fill-color:#7F7F7F;mso-style-textfill-fill-alpha:100.0%'><span style='mso-list:Ignore'>Ø<span style='font:7.0pt "Times New Roman"'> </span></span></span>why don't you post an edited version of my private replies to you to <br>this topic as well, so we have some background ? <br><br><br><span style='font-family:"Calibri","sans-serif";color:#1F497D'><o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Calibri","sans-serif";color:blue'>very good idea. Will do. <o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Calibri","sans-serif";color:blue'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-family:"Calibri","sans-serif";color:blue'>The bottom line Is this: <b>let’s fix this</b>. Let’s work together with OpenHFT to find the best “locality is a premium!” solution (I don’t think it is NIO, but I might be wrong). After we fix this, let’s demand the JGRID world fix it. We can go to 347 (BTW, OpenHFT’s Peter Lawrey is now being seated on the 347 EG) and specify that providing a transport that accommodates locality is <u>required</u> to be JGRID standard. Let’s make it better. :-) <o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Calibri","sans-serif";color:blue'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-family:"Calibri","sans-serif";color:blue'>Thank you Bela (and RedHat).<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>From:</span></b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'> Bela Ban [via Infinispan Developer List] [mailto:<a href="/user/SendEmail.jtp?type=node&node=4028929&i=0" target="_top" rel="nofollow" link="external">[hidden email]</a>] <br><b>Sent:</b> Saturday, March 1, 2014 4:30 AM<br><b>To:</b> cotton-ben<br><b>Subject:</b> Re: [infinispan-dev] Musings on ISPN/JGRPs OSI transport choices and ambitions<o:p></o:p></span></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Hi Ben, <br><br>why don't you post an edited version of my private replies to you to <br>this topic as well, so we have some background ? <br><br>In a nutshell, you want to use a zero-copy transport between processes <br>that run on the *same physical box* (e.g. shipping only pointers to <br>native shared memory between processes). Currently, using TCP or UDP <br>between processes on the same box still requires 1 or 2 copies, even <br>when a loopback device is used. <br><br>I'm interested in adding such a transport in JGroups 4, in which I plan <br>to revamp the transport to adopt an NIO based scheme, accommodating both <br>TDP and TCP. This is all still in the planning phase, but one feature <br>will be to have multiple transports running in the same stack and <br>sending messages alternatively via different transports. E.g. multicasts <br>would use UDP whereas unicasts would use TCP (by default), but this <br>could be overridden per message (with flags). <br><br>If we then had 5 physical boxes, with 20 processes on each box, for a <br>total of 100 nodes, then we could configure the stacks to run both <br>SHR_MEM and UDP: a group-wide message (a multicast) would be sent via <br>SHR_MEM *and* UDP. <br><br>The SHR_MEM transport would disseminate the message to all 20 processes <br>on the same physical box, using shared memory. The UDP transport would <br>be configured as non-loopback (IP_MULTICAST_LOOP=false), which means <br>that the message would be multicast to the other 3 physical boxes, but <br>the local multicast would be dropped. The other boxes would then use <br>SHR_MEM to disseminate the message locally to all 20 processes. <br><br>Just an idea atm, this could also be done via RELAY2, but the QoS would <br>not be the same. <br><br>I'm planning on releasing 3.5 in 6-8 weeks from now. This includes a <br>community baking phase during which I'll be working on a deep-dive <br>course on JGroups. <br><br>So a *very tentative* schedule is to start on 4.0 at the beginning of <br>summer. <br><br><br>On 28/02/14 19:16, cotton-ben wrote: <o:p></o:p></p><div><p class=MsoNormal><div class='shrinkable-quote'><br>> Hi Mircea, Manik, Bela, et. al. <br>> <br>> I want to more publicly muse on this SUBJ line. Here now, then maybe in <br>> ISPN /user/ forum, then maybe JSR-347 provider wide. I know we had a <br>> semi-private (Bela led) exchange, but I want to be more public with this <br>> conversation. <br>> <br>> Long post again. sorry. <br>> <br>> This is just on open musing. I realize this musing should not expect to be <br>> accommodated by any "oh, we got to do this in ISPN/JGRPs now!" repsonse ... <br>> there is absolutely only the most infrequent use-case that would /today/ be <br>> served by addressing this musing ... but tomorrow that /will/ be a different <br>> story. <br>> <br>> Questions:: <br>> <br>> Does the concept of ISPN/JGRPs transport between "Cluster" nodes currently <br>> depend on OSI transport layer sockets' participation(s)? <br>> <br>> In other words, if all the nodes on my "Cluster" have locality=127.0.0.1 is <br>> ISPN/JGRPs accommodating enough to use a native OS IPC choice as an <br>> intra-node transport? <br>> <br>> Or, is it true that my transport choices are always limited to just <br>> {TCP,UDP} -- independent of the participating nodes' locality (and that I <br>> am thus forced to go over an OSI loopback)? <br>> <br>> If my transport choices are only limited to {TCP,UDP} for all node locality, <br>> then I might ask that you consider additional upcoming modern Java transport <br>> options. <br>> <br>> With the ambitions of upcoming OpenJDK JEPs, that will make mainstream an <br>> API capabilty that today is only available via sun.misc.Unsafe, Java will <br>> soon have "more complete" transport options that will include all of <br>> <br>> { TCP, UDP, RDMA/SDP, IPC } <br>> <br>> Some examples of upcoming accommodating providers= <br>> <br>> 1. RDMA/SDP: via Infiniband VERBS (works today in JDK 7 on OSI physical <br>> layer IB NICs, does not work over Ethernet) <br>> 2. IPC via OpenHFT' SHM as IPC solution (will work this year) <br>> <br>> Again, I realize that these transport choices are useful today only in a <br>> very rare use case. However, should these transports be in your offering to <br>> ISPN/JGRPs customers, then ISPN/JGRPs becomes -- like all of Java has <br>> become in recent years -- increasingly more attractive to /all/ HPC Linux <br>> supercomputing use cases (not just ours). <br>> <br>> <br>> <br>> <br>> <br>> <br>> <br>> -- <br>> View this message in context: <a href="http://infinispan-developer-list.980875.n3.nabble.com/Musings-on-ISPN-JGRPs-OSI-transport-choices-and-ambitions-tp4028925.html" target="_top" rel="nofollow" link="external">http://infinispan-developer-list.980875.n3.nabble.com/Musings-on-ISPN-JGRPs-OSI-transport-choices-and-ambitions-tp4028925.html</a><br>> Sent from the Infinispan Developer List mailing list archive at Nabble.com. <br>> _______________________________________________ <br>> infinispan-dev mailing list <br>> <a href="/user/SendEmail.jtp?type=node&node=4028928&i=0" target="_top" rel="nofollow" link="external">[hidden email]</a> <br>> <a href="https://lists.jboss.org/mailman/listinfo/infinispan-dev" target="_top" rel="nofollow" link="external">https://lists.jboss.org/mailman/listinfo/infinispan-dev</a><br>> <o:p></o:p></p></div><p class=MsoNormal style='margin-bottom:12.0pt'></div>-- <br>Bela Ban, JGroups lead (<a href="http://www.jgroups.org" target="_top" rel="nofollow" link="external">http://www.jgroups.org</a>) <br>_______________________________________________ <br>infinispan-dev mailing list <br><a href="/user/SendEmail.jtp?type=node&node=4028928&i=1" target="_top" rel="nofollow" link="external">[hidden email]</a> <br><a href="https://lists.jboss.org/mailman/listinfo/infinispan-dev" target="_top" rel="nofollow" link="external">https://lists.jboss.org/mailman/listinfo/infinispan-dev</a><br><br><o:p></o:p></p><div class=MsoNormal align=center style='text-align:center'><hr size=1 width="100%" noshade style='color:#CCCCCC' align=center></div><div><div><p class=MsoNormal><b><span style='font-size:9.0pt;font-family:"Tahoma","sans-serif";color:#444444'>If you reply to this email, your message will be added to the discussion below:<o:p></o:p></span></b></p></div><p class=MsoNormal><span style='font-size:9.0pt;font-family:"Tahoma","sans-serif";color:#444444'><a href="http://infinispan-developer-list.980875.n3.nabble.com/Musings-on-ISPN-JGRPs-OSI-transport-choices-and-ambitions-tp4028925p4028928.html" target="_top" rel="nofollow" link="external">http://infinispan-developer-list.980875.n3.nabble.com/Musings-on-ISPN-JGRPs-OSI-transport-choices-and-ambitions-tp4028925p4028928.html</a> <o:p></o:p></span></p></div><div style='margin-top:4.8pt'><p class=MsoNormal style='line-height:18.0pt'><span style='font-size:8.5pt;font-family:"Tahoma","sans-serif";color:#666666'>To start a new topic under Infinispan Developer List, email <a href="/user/SendEmail.jtp?type=node&node=4028929&i=1" target="_top" rel="nofollow" link="external">[hidden email]</a> <br>To unsubscribe from Infinispan Developer List, <a href="" target="_top" rel="nofollow" link="external">click here</a>.<br><a href="http://infinispan-developer-list.980875.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml" target="_top" rel="nofollow" link="external"><span style='font-size:7.0pt;font-family:"Times New Roman","serif"'>NAML</span></a> <o:p></o:p></span></p></div></div>
<br/><hr align="left" width="300" />
View this message in context: <a href="http://infinispan-developer-list.980875.n3.nabble.com/Musings-on-ISPN-JGRPs-OSI-transport-choices-and-ambitions-tp4028925p4028929.html">RE: [infinispan-dev] Musings on ISPN/JGRPs OSI transport choices and ambitions</a><br/>
Sent from the <a href="http://infinispan-developer-list.980875.n3.nabble.com/">Infinispan Developer List mailing list archive</a> at Nabble.com.<br/>