<html>
<head>
<style><!--
.hmmessage P
{
margin:0px;
padding:0px
}
body.hmmessage
{
font-size: 10pt;
font-family:Tahoma
}
--></style>
</head>
<body class='hmmessage'>
Bela,<BR> <BR> Tried your new config. I had to leave FLUSH in for Infinispan 4.2.0. Also, it complained about the cap parameter on pbcast.STABLE, so I removed it. When I started everything up, it created 5 groups and did not appear to be merging. The merge was timing out (maybe because the FLUSH was taking too long). I increased some of the timeouts (listed below) and then it started up with three groups. I will test more tomorrow.<BR> <BR>PING timeout="10000"<BR>pbcast.GMS join_timeout="20000" merge_timeout="10000" <BR> <BR> <BR>
> Date: Tue, 22 Mar 2011 08:44:32 +0100<BR>> From: bban@redhat.com<BR>> To: infinispan-dev@lists.jboss.org<BR>> Subject: Re: [infinispan-dev] Infinispan Large Scale support<BR>> <BR>> I cross-posted this to the JGroups mailing lists [1]<BR>> <BR>> <BR>> [1] https://sourceforge.net/mail/?group_id=6081<BR>> <BR>> <BR>> On 3/22/11 2:05 AM, Dave wrote:<BR>> > I switched back to UDP today based on your feedback. Our config resembles<BR>> > the config below. Like I said we just increased sizes and timeouts. If you<BR>> > ask me why I tweaked a certain parameter my response would be that it seemed<BR>> > like a good idea based on the JGroups documentation. UDP seemed a little<BR>> > more problematic than TCP, not sure why though.<BR>> ><BR>> > <config xmlns="urn:org:jgroups"<BR>> > xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"<BR>> > xsi:schemaLocation="urn:org:jgroups file:schema/JGroups-2.8.xsd"><BR>> > <UDP<BR>> > mcast_addr="${jgroups.udp.mcast_addr:228.6.7.8}"<BR>> > mcast_port="${jgroups.udp.mcast_port:46655}"<BR>> > tos="8"<BR>> > ucast_recv_buf_size="20000000"<BR>> > ucast_send_buf_size="640000"<BR>> > mcast_recv_buf_size="25000000"<BR>> > mcast_send_buf_size="640000"<BR>> > loopback="true"<BR>> > discard_incompatible_packets="true"<BR>> > max_bundle_size="4000000"<BR>> > max_bundle_timeout="30"<BR>> > ip_ttl="${jgroups.udp.ip_ttl:2}"<BR>> > enable_bundling="true"<BR>> > enable_diagnostics="false"<BR>> ><BR>> > thread_naming_pattern="pl"<BR>> ><BR>> > thread_pool.enabled="true"<BR>> > thread_pool.min_threads="2"<BR>> > thread_pool.max_threads="30"<BR>> > thread_pool.keep_alive_time="5000"<BR>> > thread_pool.queue_enabled="true"<BR>> > thread_pool.queue_max_size="1000"<BR>> > thread_pool.rejection_policy="Discard"<BR>> ><BR>> > oob_thread_pool.enabled="true"<BR>> > oob_thread_pool.min_threads="2"<BR>> > oob_thread_pool.max_threads="30"<BR>> > oob_thread_pool.keep_alive_time="5000"<BR>> > oob_thread_pool.queue_enabled="true"<BR>> > oob_thread_pool.queue_max_size="1000"<BR>> > oob_thread_pool.rejection_policy="Discard"<BR>> > /><BR>> ><BR>> > <PING timeout="360000" num_initial_members="400"<BR>> > break_on_coord_rsp="false"/><BR>> > <MERGE2 max_interval="30000" min_interval="10000"/><BR>> > <FD_SOCK/><BR>> > <FD_ALL/><BR>> > <BARRIER /><BR>> > <pbcast.NAKACK use_stats_for_retransmission="false"<BR>> > exponential_backoff="0"<BR>> > use_mcast_xmit="true" gc_lag="0"<BR>> > retransmit_timeout="300,600,1200,2400,3600,4800"<BR>> > discard_delivered_msgs="true"/><BR>> > <UNICAST timeout="300,600,1200,2400,3600,4800"/><BR>> > <pbcast.STABLE stability_delay="1000" desired_avg_gossip="50000"<BR>> > max_bytes="1000000"/><BR>> > <pbcast.GMS print_local_addr="false" join_timeout="60000"<BR>> > view_bundling="true" use_flush_if_present="false"/><BR>> > <UFC max_credits="2000000" min_threshold="0.20"/><BR>> > <MFC max_credits="2000000" min_threshold="0.20"/><BR>> > <FRAG2 frag_size="2000000" /><BR>> > <pbcast.STREAMING_STATE_TRANSFER/><BR>> > <!--<pbcast.STATE_TRANSFER/> --><BR>> > <pbcast.FLUSH timeout="0"/><BR>> > </config><BR>> ><BR>> ><BR>> > -----Original Message-----<BR>> > From: infinispan-dev-bounces@lists.jboss.org<BR>> > [mailto:infinispan-dev-bounces@lists.jboss.org] On Behalf Of Bela Ban<BR>> > Sent: Saturday, March 19, 2011 1:15 PM<BR>> > To: infinispan-dev@lists.jboss.org<BR>> > Subject: Re: [infinispan-dev] Infinispan Large Scale support<BR>> ><BR>> > Hard to believe that TCP would be better, as TCP creates a mesh of<BR>> > connections; for 400 nodes, with every node sending, you'll have roughly<BR>> > 400*400 connections !<BR>> ><BR>> > I always had a much better experience with UDP<BR>> ><BR>> > On 3/19/11 2:37 PM, david marion wrote:<BR>> >><BR>> >> Initially yes, but I think we are getting better stability using TCP. I<BR>> > switched it back to TCP yesterday. I can post specifics of what I did in the<BR>> > TCP configuration, but the short story is I increased a lot of the timeout<BR>> > values to get it to work.<BR>> >><BR>> >> Dave Marion<BR>> >><BR>> >><BR>> >>> Date: Sat, 19 Mar 2011 10:50:54 +0100<BR>> >>> From: bban@redhat.com<BR>> >>> To: infinispan-dev@lists.jboss.org<BR>> >>> Subject: Re: [infinispan-dev] Infinispan Large Scale support<BR>> >>><BR>> >>><BR>> >>><BR>> >>> On 3/18/11 10:35 PM, Dave wrote:<BR>> >>>> Won't be able to get CR4 uploaded, policy dictates that I wait until<BR>> > final<BR>> >>>> release. However, I was able to get 431 nodes up and running as a<BR>> > replicated<BR>> >>>> cluster and 115 nodes up as a distributed cluster. For the 430 node<BR>> > cache, I<BR>> >>>> was able to get it started with no problems about 50% of the time. When<BR>> > they<BR>> >>>> formed multiple clusters they merged together only some of the time. It<BR>> >>>> really does appear to be a startup issue at this point. We have not<BR>> > pushed<BR>> >>>> it hard enough yet to see what happens at this scale under load.<BR>> >>>><BR>> >>>><BR>> >>>><BR>> >>>> Any idea when CR4 will be FINAL?<BR>> >>>><BR>> >>>> Are there any tools to help diagnose problems / performance at this<BR>> > scale (I<BR>> >>>> ended up writing my own monitor program)?<BR>> >>><BR>> >>><BR>> >>> Yes, there's probe.sh at the JGroups level. I created a JIRA to provide<BR>> >>> a sample for large clusters. You said you based your config on udp.xml,<BR>> >>> correct ?<BR>> >>><BR>> >>> [1] https://issues.jboss.org/browse/JGRP-1307<BR>> >>><BR>> >>> --<BR>> >>> Bela Ban<BR>> >>> Lead JGroups / Clustering Team<BR>> >>> JBoss<BR>> >>> _______________________________________________<BR>> >>> infinispan-dev mailing list<BR>> >>> infinispan-dev@lists.jboss.org<BR>> >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev<BR>> >> <BR>> >><BR>> >><BR>> >><BR>> >> _______________________________________________<BR>> >> infinispan-dev mailing list<BR>> >> infinispan-dev@lists.jboss.org<BR>> >> https://lists.jboss.org/mailman/listinfo/infinispan-dev<BR>> ><BR>> <BR>> -- <BR>> Bela Ban<BR>> Lead JGroups / Clustering Team<BR>> JBoss<BR>> _______________________________________________<BR>> infinispan-dev mailing list<BR>> infinispan-dev@lists.jboss.org<BR>> https://lists.jboss.org/mailman/listinfo/infinispan-dev<BR> </body>
</html>