<html>
<head>
<style><!--
.hmmessage P
{
margin:0px;
padding:0px
}
body.hmmessage
{
font-size: 10pt;
font-family:Tahoma
}
--></style>
</head>
<body class='hmmessage'>
<P style="TEXT-INDENT: 6pt; MARGIN: 0in 0in 0pt" class=MsoNormal><SPAN style="FONT-FAMILY: Arial; COLOR: navy; FONT-SIZE: 10pt">For monitoring, anything that you may have already, going through logs at this scale is tedious. I created a monitor web page that shows the following for each node:</SPAN><?xml:namespace prefix = o ns = "urn:schemas-microsoft-com:office:office" /><o:p></o:p></P>
<P style="TEXT-INDENT: 6pt; MARGIN: 0in 0in 0pt" class=MsoNormal><SPAN style="FONT-FAMILY: Arial; COLOR: navy; FONT-SIZE: 10pt"> </SPAN><o:p></o:p></P>
<P style="TEXT-INDENT: 6pt; MARGIN: 0in 0in 0pt" class=MsoNormal><SPAN style="FONT-FAMILY: Arial; COLOR: navy; FONT-SIZE: 10pt"> Node name, coordinator, transport cluster name, first heard, last heard, entries, and evictions.</SPAN><o:p></o:p></P>
<P style="TEXT-INDENT: 6pt; MARGIN: 0in 0in 0pt" class=MsoNormal><SPAN style="FONT-FAMILY: Arial; COLOR: navy; FONT-SIZE: 10pt"> </SPAN><o:p></o:p></P>
<P style="TEXT-INDENT: 6pt; MARGIN: 0in 0in 0pt" class=MsoNormal><SPAN style="FONT-FAMILY: Arial; COLOR: navy; FONT-SIZE: 10pt">This will tell me right away how many nodes are up and if they are all using the same coordinator. I am running Infinispan in an embedded fashion, so it�s my code that interrogates each cache and reports the information to the monitor server. It�s actually been pretty handy because we can see the nodes that either have not started or have not formed one cluster.</SPAN><o:p></o:p></P>
<P style="TEXT-INDENT: 6pt; MARGIN: 0in 0in 0pt" class=MsoNormal><SPAN style="FONT-FAMILY: Arial; COLOR: navy; FONT-SIZE: 10pt"> </SPAN><o:p></o:p></P>
<P style="TEXT-INDENT: 6pt; MARGIN: 0in 0in 0pt" class=MsoNormal><SPAN style="FONT-FAMILY: Arial; COLOR: navy; FONT-SIZE: 10pt">Do you have a number for the JIRA issue?</SPAN><o:p></o:p></P>
<P style="TEXT-INDENT: 6pt; MARGIN: 0in 0in 0pt" class=MsoNormal><SPAN style="FONT-FAMILY: Arial; COLOR: navy; FONT-SIZE: 10pt"> </SPAN><o:p></o:p></P>
<P style="TEXT-INDENT: 6pt; MARGIN: 0in 0in 0pt" class=MsoNormal><SPAN style="FONT-FAMILY: Arial; COLOR: navy; FONT-SIZE: 10pt">Dave Marion</SPAN><o:p></o:p></P>
<BR> <BR>
<HR id=stopSpelling>
From: manik@jboss.org<BR>Date: Sat, 19 Mar 2011 10:07:32 +0000<BR>To: infinispan-dev@lists.jboss.org<BR>Subject: Re: [infinispan-dev] Infinispan Large Scale support<BR><BR>
<META name=Generator content="Microsoft SafeHTML"><BASE><BR>
<DIV>
<DIV>On 18 Mar 2011, at 21:35, Dave wrote:</DIV><BR class=ecxApple-interchange-newline>
<BLOCKQUOTE><SPAN style="TEXT-TRANSFORM: none; TEXT-INDENT: 0px; BORDER-COLLAPSE: separate; FONT: medium Helvetica; WHITE-SPACE: normal; LETTER-SPACING: normal; WORD-SPACING: 0px" class=ecxApple-style-span>
<DIV lang=EN-US>
<DIV style="page: Section1" class=ecxSection1>
<DIV style="FONT-FAMILY: 'Times New Roman'; MARGIN-BOTTOM: 0pt; MARGIN-LEFT: 0in; FONT-SIZE: 12pt; MARGIN-RIGHT: 0in"><FONT color=navy size=2 face=Arial><SPAN style="FONT-FAMILY: Arial; COLOR: navy; FONT-SIZE: 10pt"> Won�t be able to get CR4 uploaded, policy dictates that I wait until final release. However, I was able to get 431 nodes up and running as a replicated cluster and 115 nodes up as a distributed cluster. For the 430 node cache, I was able to get it started with no problems about 50% of the time. When they formed multiple clusters they merged together only some of the time. It really does appear to be a startup issue at thi</SPAN></FONT><FONT color=navy size=2 face=Arial><SPAN style="FONT-FAMILY: Arial; COLOR: navy; FONT-SIZE: 10pt">s p</SPAN></FONT><FONT color=navy size=2 face=Arial><SPAN style="FONT-FAMILY: Arial; COLOR: navy; FONT-SIZE: 10pt">oint. We have not pushed it hard enough yet to see what happens at this scale under load.</SPAN></FONT></DIV>
<P style="FONT-FAMILY: 'Times New Roman'; MARGIN-BOTTOM: 0pt; MARGIN-LEFT: 0in; FONT-SIZE: 12pt; MARGIN-RIGHT: 0in" class=ecxMsoNormal><FONT color=navy size=2 face=Arial><SPAN style="FONT-FAMILY: Arial; COLOR: navy; FONT-SIZE: 10pt"> </SPAN></FONT></P>
<DIV style="FONT-FAMILY: 'Times New Roman'; MARGIN-BOTTOM: 0pt; MARGIN-LEFT: 0in; FONT-SIZE: 12pt; MARGIN-RIGHT: 0in"><FONT color=navy size=2 face=Arial><SPAN style="FONT-FAMILY: Arial; COLOR: navy; FONT-SIZE: 10pt">Any idea when CR4 will be FINAL?</SPAN></FONT></DIV></DIV></DIV></SPAN></BLOCKQUOTE>
<DIV><BR></DIV>
<DIV>Hopefully some time next week.</DIV>
<DIV><BR></DIV>
<DIV>I have documented the system property on the JIRA.</DIV><BR>
<BLOCKQUOTE><SPAN style="TEXT-TRANSFORM: none; TEXT-INDENT: 0px; BORDER-COLLAPSE: separate; FONT: medium Helvetica; WHITE-SPACE: normal; LETTER-SPACING: normal; WORD-SPACING: 0px" class=ecxApple-style-span>
<DIV lang=EN-US>
<DIV style="page: Section1" class=ecxSection1>
<DIV style="FONT-FAMILY: 'Times New Roman'; MARGIN-BOTTOM: 0pt; MARGIN-LEFT: 0in; FONT-SIZE: 12pt; MARGIN-RIGHT: 0in"><FONT color=navy size=2 face=Arial><SPAN style="FONT-FAMILY: Arial; COLOR: navy; FONT-SIZE: 10pt">Are there any tools to help diagnose problems / performance at this scale (I ended up writing my own monitor program)?</SPAN></FONT></DIV></DIV></DIV></SPAN></BLOCKQUOTE>
<DIV><BR></DIV>
<DIV>Specifically what tools are you after?</DIV>
<DIV><BR></DIV>
<DIV>Cheers</DIV>
<DIV>Manik</DIV>
<DIV><BR></DIV>
<DIV>--</DIV></DIV>
<DIV><SPAN style="TEXT-TRANSFORM: none; TEXT-INDENT: 0px; BORDER-COLLAPSE: separate; FONT: medium Helvetica; WHITE-SPACE: normal; LETTER-SPACING: normal; COLOR: rgb(0,0,0); WORD-SPACING: 0px" class=ecxApple-style-span>
<DIV>
<DIV>Manik Surtani</DIV>
<DIV><A href="mailto:manik@jboss.org">manik@jboss.org</A></DIV>
<DIV><A href="http://twitter.com/maniksurtani" target=_blank>twitter.com/maniksurtani</A></DIV>
<DIV><BR></DIV>
<DIV>Lead, Infinispan</DIV>
<DIV><A href="http://www.infinispan.org/" target=_blank>http://www.infinispan.org</A></DIV>
<DIV><BR></DIV></DIV></SPAN><BR class=ecxApple-interchange-newline></DIV><BR><BR>_______________________________________________ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev </body>
</html>