<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=us-ascii"><meta name=Generator content="Microsoft Word 14 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:Helvetica;
panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
span.apple-style-span
{mso-style-name:apple-style-span;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:1306735148;
mso-list-type:hybrid;
mso-list-template-ids:333203498 657734074 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;}
@list l0:level1
{mso-level-number-format:bullet;
mso-level-text:-;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Calibri","sans-serif";
mso-fareast-font-family:Calibri;}
@list l0:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l0:level3
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
@list l0:level4
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Symbol;}
@list l0:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l0:level6
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
@list l0:level7
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Symbol;}
@list l0:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l0:level9
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-US link=blue vlink=purple><div class=WordSection1><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Hi all,<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>There are a couple of reasons: <o:p></o:p></span></p><p class=MsoListParagraph style='text-indent:-.25in;mso-list:l0 level1 lfo1'><![if !supportLists]><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><span style='mso-list:Ignore'>-<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>I may not want a task failover policy at all, and the current API kind of obscures this. At the very least, this is different than the 5.1 behavior. <o:p></o:p></span></p><p class=MsoListParagraph style='text-indent:-.25in;mso-list:l0 level1 lfo1'><![if !supportLists]><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><span style='mso-list:Ignore'>-<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>I specify keys to a task to represent the key set that will be pessimistically acquired to eliminate RPCs. (there’s some Group API magic and taking advantage of the new SyncConsistentHash). A random policy defeats this purpose. Thus if I do want a task failover policy, I want one that allows the task to be retried on the node that owns the representative keys.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Incidentally, the failover policy obscures the original reason the task failed. The exception thrown to the calling node is a FailoverException. The original reason is about 4 levels deep. Example:<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>java.util.concurrent.ExecutionException: Failover execution failed<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'> at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.failoverExecution(DefaultExecutorService.java:855)<o:p></o:p></span></p><p class=MsoNormal style='text-indent:.5in'><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>…<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Caused by: java.lang.Exception: Failover execution failed<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'> ... 45 more<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Caused by: java.util.concurrent.ExecutionException: Failover execution failed<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'> at org.infinispan.distexec.DefaultExecutorService$DistributedTaskPart.failoverExecution(DefaultExecutorService.java:852)<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'> ... 44 more<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Caused by: java.lang.Exception: Failover execution failed<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'> ... 45 more<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Caused by: java.util.concurrent.ExecutionException: org.infinispan.util.concurrent.TimeoutException: Unable to acquire lock after [5 seconds] on key [ServiceGroupKey[edgeDeviceId=1,serviceGroupNo=101]] for requestor [GlobalTransaction:<east-dg02-61087(east)>:1203:remote]! Lock held by [GlobalTransaction:<east-dg02-61087(east)>:1198:remote]<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'> ... 43 more<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Caused by: org.infinispan.util.concurrent.TimeoutException: Unable to acquire lock after [5 seconds] on key [ServiceGroupKey[edgeDeviceId=1,serviceGroupNo=101]] for requestor [GlobalTransaction:<east-dg02-61087(east)>:1203:remote]! Lock held by [GlobalTransaction:<east-dg02-61087(east)>:1198:remote]<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'> at org.infinispan.util.concurrent.locks.LockManagerImpl.lock(LockManagerImpl.java:217)<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'> at org.infinispan.util.concurrent.locks.LockManagerImpl.acquireLock(LockManagerImpl.java:190)<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>I really need the original exception for processing/reporting to my analytics engine.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Thanks,<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Erik<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><div><div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in'><p class=MsoNormal><b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>From:</span></b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'> Mircea Markus [mailto:mircea.markus@jboss.com] <br><b>Sent:</b> Tuesday, October 23, 2012 11:40 AM<br><b>To:</b> infinispan -Dev List<br><b>Cc:</b> Erik Salter<br><b>Subject:</b> Re: [infinispan-dev] Dist.exec failover<o:p></o:p></span></p></div></div><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><o:p> </o:p></p><div><div><p class=MsoNormal>On 22 Oct 2012, at 19:48, Vladimir Blagojevic wrote:<o:p></o:p></p></div><p class=MsoNormal><br><br><o:p></o:p></p><div><p class=MsoNormal>Hey guys,<br><br>Erik noted that we should by default have no failover policy installed <br>rather the default random policy we currently have.<o:p></o:p></p></div><div><p class=MsoNormal>The random policy tries to re-run the task in case of a cluster failure. I imagine the user would do the same, so not sure why not to add it..<o:p></o:p></p></div><div><p class=MsoNormal>Erik, mind comment about the cons of having this failover in place? <o:p></o:p></p></div><div><p class=MsoNormal> <o:p></o:p></p></div><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><div><p class=MsoNormal>Also he noted that <br>keys are never supplied to failover policy and they might be important <br>when it comes to a deciding where to dispatch the failed over task.<o:p></o:p></p></div></blockquote><p class=MsoNormal>The main reason for passing the keys is in order to calculate the locality of the task based on consistent hash. We don't have a reference to the consistent hash in the DistributedTaskFailoverPolicy, so not sure this would be useful as it is.<o:p></o:p></p></div><div><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><div><p class=MsoNormal><br>Our current interface is :<br><br>public interface DistributedTaskFailoverPolicy {<br>Address failover(Address failedExecution, List<Address> <br>executionCandidates, Exception cause);<br>}<br><br>Rather than adding yet another parameter here maybe we should make a <br>simple container class<br><br>public class FailoverContext {<br><br>Address failedExecution;<br>List<Address> executionCandidates;<br>Exception cause;<br>List<Object> inputKeys;<br>}<br><br>and have<br><br><br>public interface DistributedTaskFailoverPolicy {<br>Address failover(FailoverContext context);<br>}<br><br>WDYT?<br><br>Regards,<br>Vladimir<br>_______________________________________________<br>infinispan-dev mailing list<br><a href="mailto:infinispan-dev@lists.jboss.org">infinispan-dev@lists.jboss.org</a><br><a href="https://lists.jboss.org/mailman/listinfo/infinispan-dev">https://lists.jboss.org/mailman/listinfo/infinispan-dev</a><o:p></o:p></p></div></blockquote></div><p class=MsoNormal><o:p> </o:p></p><div><div><div><div><div><div><p class=MsoNormal><span style='font-size:13.5pt;font-family:"Helvetica","sans-serif";color:black'>Cheers,<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:13.5pt;font-family:"Helvetica","sans-serif";color:black'>-- <br>Mircea Markus<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:13.5pt;font-family:"Helvetica","sans-serif";color:black'>Infinispan lead (<a href="http://www.infinispan.org">www.infinispan.org</a>)<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:13.5pt;font-family:"Helvetica","sans-serif";color:black'><o:p> </o:p></span></p></div></div></div></div></div><p class=MsoNormal><span style='font-size:13.5pt;font-family:"Helvetica","sans-serif";color:black'><br><br></span><o:p></o:p></p></div><p class=MsoNormal><o:p> </o:p></p></div></body></html>