<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=us-ascii"><meta name=Generator content="Microsoft Word 14 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:windowtext;}
span.apple-converted-space
{mso-style-name:apple-converted-space;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:484471322;
mso-list-type:hybrid;
mso-list-template-ids:-255670646 -1813315950 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;}
@list l0:level1
{mso-level-number-format:bullet;
mso-level-text:"";
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Calibri","sans-serif";
mso-fareast-font-family:Calibri;}
@list l0:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l0:level3
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
@list l0:level4
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Symbol;}
@list l0:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l0:level6
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
@list l0:level7
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Symbol;}
@list l0:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l0:level9
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
@list l1
{mso-list-id:696589408;
mso-list-type:hybrid;
mso-list-template-ids:-1296906050 1808138436 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;}
@list l1:level1
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Calibri","sans-serif";
color:#1F497D;}
@list l1:level2
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l1:level3
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l1:level4
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l1:level5
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l1:level6
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l1:level7
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l1:level8
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l1:level9
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l2
{mso-list-id:1265724002;
mso-list-type:hybrid;
mso-list-template-ids:-147956062 67698703 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;}
@list l2:level1
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l2:level2
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l2:level3
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l2:level4
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l2:level5
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l2:level6
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l2:level7
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l2:level8
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l2:level9
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l3
{mso-list-id:1546212723;
mso-list-type:hybrid;
mso-list-template-ids:-1447765666 505949572 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;}
@list l3:level1
{mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:0in;
text-indent:-.25in;
color:windowtext;}
@list l3:level2
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:.5in;
text-indent:-.25in;}
@list l3:level3
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
margin-left:1.0in;
text-indent:-9.0pt;}
@list l3:level4
{mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:1.5in;
text-indent:-.25in;}
@list l3:level5
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:2.0in;
text-indent:-.25in;}
@list l3:level6
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
margin-left:2.5in;
text-indent:-9.0pt;}
@list l3:level7
{mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:3.0in;
text-indent:-.25in;}
@list l3:level8
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:3.5in;
text-indent:-.25in;}
@list l3:level9
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
margin-left:4.0in;
text-indent:-9.0pt;}
@list l4
{mso-list-id:2058039919;
mso-list-type:hybrid;
mso-list-template-ids:-1818327778 67698703 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;}
@list l4:level1
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l4:level2
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l4:level3
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l4:level4
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l4:level5
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l4:level6
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l4:level7
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l4:level8
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l4:level9
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-US link=blue vlink=purple><div class=WordSection1><p class=MsoNormal>Hi all,<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>I’ve spent quite a bit of time with the existing XSite implementation, getting my solution to run in multiple data centers. I’ve been talking with Mircea and Bela on and off over the past few weeks, but since this affects the community and potentially commercial customers, I wanted to share with a wider audience.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal> The problems I see are as follows: <o:p></o:p></p><p class=MsoNormal> <o:p></o:p></p><p class=MsoListParagraph style='text-indent:-.25in;mso-list:l2 level1 lfo5'><![if !supportLists]><span style='mso-list:Ignore'>1.<span style='font:7.0pt "Times New Roman"'> </span></span><![endif]>A bridge end will see all the traffic – sending and receiving – for all nodes within a cluster. <o:p></o:p></p><p class=MsoListParagraph style='text-indent:-.25in;mso-list:l2 level1 lfo5'><![if !supportLists]><span style='mso-list:Ignore'>2.<span style='font:7.0pt "Times New Roman"'> </span></span><![endif]>The bridge end of a site will apply each change with the bridge end as the transaction originator.<o:p></o:p></p><p class=MsoNormal> <o:p></o:p></p><p class=MsoNormal>In my deployment, this can be three sites backing up their data to the other two. So for 3 sites of 12 nodes each, a single bridge end will see all 36 nodes worth of traffic. This breaks linear scalability. In my QA’s testing, a 3 DC cluster of 6 nodes is about 1/10 the throughput of a single cluster.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>I think I-RAC solves some of the problem, like the reliable sending of data, but it doesn’t really help with performance in high throughput cases. (Note: FWIW, my apps do about a 8-12:1 read/write ratio). <o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>So I’ve prototyped the following:<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoListParagraph style='text-indent:-.25in;mso-list:l1 level1 lfo2'><![if !supportLists]><span style='color:#1F497D'><span style='mso-list:Ignore'>1.<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]>Load-balanced applying the changes from a remote SiteMaster across all local nodes in a cluster. The basics are that there is still a single SiteMaster (thereby not breaking the existing JGroups model). This is okay, since it’s the same bandwidth pipe, and as long as there is no unmarshalling, it’s a simple buffer copy. The difference is that the messages are now forwarded to other nodes in the local cluster and delivered to the ISPN layer there for unmarshalling and data application. Note that this does NOT break XSite synchronous replication, as I’m still preserving the originating site.<o:p></o:p></p><p class=MsoNormal style='text-indent:-.25in'><o:p> </o:p></p><p class=MsoListParagraph style='text-indent:-.25in;mso-list:l1 level1 lfo2'><![if !supportLists]><span style='color:#1F497D'><span style='mso-list:Ignore'>2.<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]>I also needed more intelligent application of the data that is replicated. My local cluster will save data to 8-9 caches that need to be replicated. Instead of replicating data on cache boundaries, I consolidated the data to only replicate an aggregate object. In turn, I have a custom BackupReceiver implementation that takes this object and expands it into the requisite data for the 8-9 caches. Since these caches are a mixture of optimistic and pessimistic modes, I made liberal use of the Distributed Executor framework to execute on the data owner for any pessimistic caches. <o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>The early results are very promising, especially WRT async replication. (Sync replication just sucks – trying to design that out of the system)<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>There are a few changes made to support a custom BackupReceiver implementation [1]. There are some other ideas I had floating around in my head, but it’s such a fine line between clever and stupid.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoListParagraph style='text-indent:-.25in;mso-list:l4 level1 lfo3'><![if !supportLists]><span style='mso-list:Ignore'>1.<span style='font:7.0pt "Times New Roman"'> </span></span><![endif]>I’m prototyping an option where the SiteMaster would load-balance among the other cluster members, but exclude itself. Preliminary testing shows that this really only helps when the cluster size > numOwners + 1.<o:p></o:p></p><p class=MsoListParagraph style='text-indent:-.25in;mso-list:l4 level1 lfo3'><![if !supportLists]><span style='mso-list:Ignore'>2.<span style='font:7.0pt "Times New Roman"'> </span></span><![endif]>I think the staggered get work will be vital in these deployments. Mircea had the idea of suppressing reads on the SiteMaster node.<o:p></o:p></p><p class=MsoListParagraph style='text-indent:-.25in;mso-list:l4 level1 lfo3'><![if !supportLists]><span style='mso-list:Ignore'>3.<span style='font:7.0pt "Times New Roman"'> </span></span><![endif]>In the case of numerous modifications, I’ve seen them processed “out of order”. This is where I-RAC, with batched modifications, can mitigate this. (Maybe Total Order?) <o:p></o:p></p><p class=MsoListParagraph style='text-indent:-.25in;mso-list:l4 level1 lfo3'><![if !supportLists]><span style='mso-list:Ignore'>4.<span style='font:7.0pt "Times New Roman"'> </span></span><![endif]>In the case of single-key modifications, I was playing around with adding a int header to the RELAY2 message to give any application a “hint” about the hash code of the key. When ISPN received the data, there would be an initial deserialization penalty as ISPN would need to map the hash code to the address of the data owner for that cache (in 5.2 onwards, there is no guarantee that the segments for caches in a cache manager will map as in 5.1.x). This would build a “routing table” of sorts to RELAY2, so if it sees this key, it’ll know to route it to the owner directly. (And on a view change, this table would be cleared) Really, though, this was a thought for improving synchronous replication performance.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Any other thoughts? Feedback?<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>[1] <a href="https://github.com/an1310/infinispan/tree/t_receiver"><span style='color:windowtext'>https://github.com/an1310/infinispan/tree/t_receiver</span></a><o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Regards,<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Erik<o:p></o:p></p></div></body></html>