<html>
<head>
<style><!--
.hmmessage P
{
margin:0px;
padding:0px
}
body.hmmessage
{
font-size: 10pt;
font-family:Tahoma
}
--></style>
</head>
<body class='hmmessage'>
<P style="TEXT-INDENT: 6pt; MARGIN: 0in 0in 0pt" class=MsoNormal><SPAN style="FONT-FAMILY: Arial; COLOR: navy; FONT-SIZE: 10pt">For monitoring, anything that you may have already, going through logs at this scale is tedious. I created a monitor web page that shows the following for each node:</SPAN><?xml:namespace prefix = o ns = "urn:schemas-microsoft-com:office:office" /><o:p></o:p></P>
<P style="TEXT-INDENT: 6pt; MARGIN: 0in 0in 0pt" class=MsoNormal><SPAN style="FONT-FAMILY: Arial; COLOR: navy; FONT-SIZE: 10pt"> </SPAN><o:p></o:p></P>
<P style="TEXT-INDENT: 6pt; MARGIN: 0in 0in 0pt" class=MsoNormal><SPAN style="FONT-FAMILY: Arial; COLOR: navy; FONT-SIZE: 10pt"> Node name, coordinator, transport cluster name, first heard, last heard, entries, and evictions.</SPAN><o:p></o:p></P>
<P style="TEXT-INDENT: 6pt; MARGIN: 0in 0in 0pt" class=MsoNormal><SPAN style="FONT-FAMILY: Arial; COLOR: navy; FONT-SIZE: 10pt"> </SPAN><o:p></o:p></P>
<P style="TEXT-INDENT: 6pt; MARGIN: 0in 0in 0pt" class=MsoNormal><SPAN style="FONT-FAMILY: Arial; COLOR: navy; FONT-SIZE: 10pt">This will tell me right away how many nodes are up and if they are all using the same coordinator. I am running Infinispan in an embedded fashion, so it’s my code that interrogates each cache and reports the information to the monitor server. It’s actually been pretty handy because we can see the nodes that either have not started or have not formed one cluster.</SPAN><o:p></o:p></P>
<P style="TEXT-INDENT: 6pt; MARGIN: 0in 0in 0pt" class=MsoNormal><SPAN style="FONT-FAMILY: Arial; COLOR: navy; FONT-SIZE: 10pt"> </SPAN><o:p></o:p></P>
<P style="TEXT-INDENT: 6pt; MARGIN: 0in 0in 0pt" class=MsoNormal><SPAN style="FONT-FAMILY: Arial; COLOR: navy; FONT-SIZE: 10pt">Do you have a number for the JIRA issue?</SPAN><o:p></o:p></P>
<P style="TEXT-INDENT: 6pt; MARGIN: 0in 0in 0pt" class=MsoNormal><SPAN style="FONT-FAMILY: Arial; COLOR: navy; FONT-SIZE: 10pt"> </SPAN><o:p></o:p></P>
<P style="TEXT-INDENT: 6pt; MARGIN: 0in 0in 0pt" class=MsoNormal><SPAN style="FONT-FAMILY: Arial; COLOR: navy; FONT-SIZE: 10pt">Dave Marion</SPAN><o:p></o:p></P>
<BR> <BR>
<HR id=stopSpelling>
From: manik@jboss.org<BR>Date: Sat, 19 Mar 2011 10:07:32 +0000<BR>To: infinispan-dev@lists.jboss.org<BR>Subject: Re: [infinispan-dev] Infinispan Large Scale support<BR><BR>
<META name=Generator content="Microsoft SafeHTML"><BASE><BR>
<DIV>
<DIV>On 18 Mar 2011, at 21:35, Dave wrote:</DIV><BR class=ecxApple-interchange-newline>
<BLOCKQUOTE><SPAN style="TEXT-TRANSFORM: none; TEXT-INDENT: 0px; BORDER-COLLAPSE: separate; FONT: medium Helvetica; WHITE-SPACE: normal; LETTER-SPACING: normal; WORD-SPACING: 0px" class=ecxApple-style-span>
<DIV lang=EN-US>
<DIV style="page: Section1" class=ecxSection1>
<DIV style="FONT-FAMILY: 'Times New Roman'; MARGIN-BOTTOM: 0pt; MARGIN-LEFT: 0in; FONT-SIZE: 12pt; MARGIN-RIGHT: 0in"><FONT color=navy size=2 face=Arial><SPAN style="FONT-FAMILY: Arial; COLOR: navy; FONT-SIZE: 10pt"> Won’t be able to get CR4 uploaded, policy dictates that I wait until final release. However, I was able to get 431 nodes up and running as a replicated cluster and 115 nodes up as a distributed cluster. For the 430 node cache, I was able to get it started with no problems about 50% of the time. When they formed multiple clusters they merged together only some of the time. It really does appear to be a startup issue at thi</SPAN></FONT><FONT color=navy size=2 face=Arial><SPAN style="FONT-FAMILY: Arial; COLOR: navy; FONT-SIZE: 10pt">s p</SPAN></FONT><FONT color=navy size=2 face=Arial><SPAN style="FONT-FAMILY: Arial; COLOR: navy; FONT-SIZE: 10pt">oint. We have not pushed it hard enough yet to see what happens at this scale under load.</SPAN></FONT></DIV>
<P style="FONT-FAMILY: 'Times New Roman'; MARGIN-BOTTOM: 0pt; MARGIN-LEFT: 0in; FONT-SIZE: 12pt; MARGIN-RIGHT: 0in" class=ecxMsoNormal><FONT color=navy size=2 face=Arial><SPAN style="FONT-FAMILY: Arial; COLOR: navy; FONT-SIZE: 10pt"> </SPAN></FONT></P>
<DIV style="FONT-FAMILY: 'Times New Roman'; MARGIN-BOTTOM: 0pt; MARGIN-LEFT: 0in; FONT-SIZE: 12pt; MARGIN-RIGHT: 0in"><FONT color=navy size=2 face=Arial><SPAN style="FONT-FAMILY: Arial; COLOR: navy; FONT-SIZE: 10pt">Any idea when CR4 will be FINAL?</SPAN></FONT></DIV></DIV></DIV></SPAN></BLOCKQUOTE>
<DIV><BR></DIV>
<DIV>Hopefully some time next week.</DIV>
<DIV><BR></DIV>
<DIV>I have documented the system property on the JIRA.</DIV><BR>
<BLOCKQUOTE><SPAN style="TEXT-TRANSFORM: none; TEXT-INDENT: 0px; BORDER-COLLAPSE: separate; FONT: medium Helvetica; WHITE-SPACE: normal; LETTER-SPACING: normal; WORD-SPACING: 0px" class=ecxApple-style-span>
<DIV lang=EN-US>
<DIV style="page: Section1" class=ecxSection1>
<DIV style="FONT-FAMILY: 'Times New Roman'; MARGIN-BOTTOM: 0pt; MARGIN-LEFT: 0in; FONT-SIZE: 12pt; MARGIN-RIGHT: 0in"><FONT color=navy size=2 face=Arial><SPAN style="FONT-FAMILY: Arial; COLOR: navy; FONT-SIZE: 10pt">Are there any tools to help diagnose problems / performance at this scale (I ended up writing my own monitor program)?</SPAN></FONT></DIV></DIV></DIV></SPAN></BLOCKQUOTE>
<DIV><BR></DIV>
<DIV>Specifically what tools are you after?</DIV>
<DIV><BR></DIV>
<DIV>Cheers</DIV>
<DIV>Manik</DIV>
<DIV><BR></DIV>
<DIV>--</DIV></DIV>
<DIV><SPAN style="TEXT-TRANSFORM: none; TEXT-INDENT: 0px; BORDER-COLLAPSE: separate; FONT: medium Helvetica; WHITE-SPACE: normal; LETTER-SPACING: normal; COLOR: rgb(0,0,0); WORD-SPACING: 0px" class=ecxApple-style-span>
<DIV>
<DIV>Manik Surtani</DIV>
<DIV><A href="mailto:manik@jboss.org">manik@jboss.org</A></DIV>
<DIV><A href="http://twitter.com/maniksurtani" target=_blank>twitter.com/maniksurtani</A></DIV>
<DIV><BR></DIV>
<DIV>Lead, Infinispan</DIV>
<DIV><A href="http://www.infinispan.org/" target=_blank>http://www.infinispan.org</A></DIV>
<DIV><BR></DIV></DIV></SPAN><BR class=ecxApple-interchange-newline></DIV><BR><BR>_______________________________________________ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev </body>
</html>