[jboss-jira] [JBoss JIRA] Closed: (JGRP-343) Unable to bring additional node to a cluster due to WARN ClientGmsImpl:331 - there was more than 1 candidate for coord and WARN ClientGmsImpl:118 - join(10.20.111.1:1362) failed, retrying
Bela Ban (JIRA)
jira-events at jboss.com
Fri Oct 20 04:23:42 EDT 2006
[ http://jira.jboss.com/jira/browse/JGRP-343?page=all ]
Bela Ban closed JGRP-343.
-------------------------
Resolution: Incomplete Description
> Unable to bring additional node to a cluster due to WARN ClientGmsImpl:331 - there was more than 1 candidate for coord and WARN ClientGmsImpl:118 - join(10.20.111.1:1362) failed, retrying
> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: JGRP-343
> URL: http://jira.jboss.com/jira/browse/JGRP-343
> Project: JGroups
> Issue Type: Bug
> Affects Versions: 2.2.8
> Environment: Win2k, Linux RedHat
> Linux qacluster01 2.4.21-32.0.1.ELsmp #1 SMP Tue May 17 17:52:23 EDT 2005 i686 i686 i386 GNU/Linux
> Reporter: Maciek Kolesnik
> Assigned To: Bela Ban
>
> We have an environment that consists of approx 20-30 nodes all running Hibernate 3.1.1->OSCache->Jgroups 2.2.8
> In some cases the state of the cluster becomes inconsitent due to multiple nodes being rebooted at random times and this results in some nodes thnking that Coordinator is NODE-X and other nodes thinking that Coordinator is NODE-Y. It is actually very easy to repsoduc it in this 20-node environment by rebooting the coordinator node.
> The confused nodes never recover from the wrong state even when NODE-X goes off-line. The only way to correct is to reboot all ~20 nodes which really sucks
> During the confused cluster state i am unable to bring additional nodes up - the JOIN error keeps scrolling and they never become part of the cluster:
> 19:20:04,500 WARN ClientGmsImpl:118 - join(10.20.111.1:1362) failed, retrying
> ...
> With debug on we were able to determine that the reason for it is the following:
> 16:51:56,203 WARN ClientGmsImpl:331 - there was more than 1 candidate for coord
> inator: {10.40.10.52:4775=1, 10.20.110.1:2715=1}
> This was when node 10.20.110.1 has been dead for at least an hour.
> The state of the cluster (each nodes' oppinion on who should be the coordinator is nicely reported here:
> 16:52:06,375 DEBUG PingWaiter:95 - clearing old responses: [[own_addr=10.20.111.
> 1:1117, coord_addr=10.20.111.1:1117, is_server=false], [own_addr=10.20.111.3:651
> 80, coord_addr=10.40.10.52:4775, is_server=true], [own_addr=10.20.110.2:56053, c
> oord_addr=10.20.110.1:2715, is_server=true], [own_addr=10.20.110.3:45282, coord_
> addr=10.40.10.52:4775, is_server=true], [own_addr=10.20.111.2:1620, coord_addr=1
> 0.20.110.1:2715, is_server=true], [own_addr=10.40.10.51:3426, coord_addr=10.20.1
> 10.1:2715, is_server=true], [own_addr=10.40.10.52:4775, coord_addr=10.20.110.1:2
> 715, is_server=true], [own_addr=10.40.121.3:4249, coord_addr=10.20.110.1:2715, i
> s_server=true], [own_addr=10.40.125.3:1739, coord_addr=10.20.110.1:2715, is_serv
> er=true], [own_addr=10.40.126.2:4228, coord_addr=10.20.110.1:2715, is_server=tru
> e], [own_addr=10.40.124.1:64677, coord_addr=10.40.124.1:64677, is_server=false],
> [own_addr=10.40.124.2:31634, coord_addr=10.20.110.1:2715, is_server=true], [own
> _addr=10.40.121.2:1403, coord_addr=10.20.110.1:2715, is_server=true], [own_addr=
> 10.40.120.3:4838, coord_addr=10.20.110.1:2715, is_server=true], [own_addr=10.40.
> 126.3:4204, coord_addr=10.20.110.1:2715, is_server=true], [own_addr=10.40.126.1:
> 58260, coord_addr=10.20.110.1:2715, is_server=true], [own_addr=10.40.125.2:3377,
> coord_addr=10.20.110.1:2715, is_server=true], [own_addr=10.40.124.4:1764, coord
> _addr=10.20.110.1:2715, is_server=true], [own_addr=10.40.121.4:1832, coord_addr=
> 10.20.110.1:2715, is_server=true], [own_addr=10.40.125.4:2799, coord_addr=10.20.
> 110.1:2715, is_server=true], [own_addr=10.40.120.4:3696, coord_addr=10.20.110.1:
> 2715, is_server=true], [own_addr=10.40.120.2:2013, coord_addr=10.20.110.1:2715,
> is_server=true], [own_addr=10.40.126.4:21141, coord_addr=10.20.110.1:2715, is_se
> rver=true], [own_addr=10.40.124.3:15988, coord_addr=10.20.110.1:2715, is_server=
> true], [own_addr=10.40.125.1:3612, coord_addr=10.20.110.1:2715, is_server=true],
> [own_addr=10.40.124.1:64720, coord_addr=10.40.124.1:64720, is_server=false], [o
> wn_addr=10.40.121.1:3810, coord_addr=10.20.110.1:2715, is_server=true]]
> Majority of the nodes think that our DEAD node should be the Coordinator.
> And finally here is the full debug trace from an attempt to bring a new node into the cluster:
> D:\sbroot\r6.1.0.0\Bin>startEmulator.bat
> 16:51:49,578 DEBUG ClassConfigurator:97 - mapping is:
> 1: class org.jgroups.stack.IpAddress
> 2: class org.jgroups.protocols.CAUSAL$CausalHeader
> 3: class org.jgroups.protocols.FD$FdHeader
> 4: class org.jgroups.protocols.FD_PID$FdHeader
> 5: class org.jgroups.protocols.FD_PROB$FdHeader
> 6: class org.jgroups.protocols.FD_SOCK$FdHeader
> 7: class org.jgroups.protocols.FragHeader
> 8: class org.jgroups.protocols.MERGE$MergeHeader
> 9: class org.jgroups.protocols.NakAckHeader
> 10: class org.jgroups.protocols.PARTITIONER$PartitionerHeader
> 11: class org.jgroups.protocols.PerfHeader
> 12: class org.jgroups.protocols.PIGGYBACK$PiggybackHeader
> 13: class org.jgroups.protocols.PingHeader
> 14: class org.jgroups.protocols.TcpHeader
> 15: class org.jgroups.protocols.TOTAL$Header
> 16: class org.jgroups.protocols.TOTAL_OLD$TotalHeader
> 17: class org.jgroups.protocols.TOTAL_TOKEN$TotalTokenHeader
> 18: class org.jgroups.protocols.TOTAL_TOKEN$RingTokenHeader
> 19: class org.jgroups.protocols.TunnelHeader
> 20: class org.jgroups.protocols.UdpHeader
> 21: class org.jgroups.protocols.UNICAST$UnicastHeader
> 22: class org.jgroups.protocols.VERIFY_SUSPECT$VerifyHeader
> 23: class org.jgroups.protocols.WANPIPE$WanPipeHeader
> 24: class org.jgroups.protocols.pbcast.GMS$GmsHeader
> 25: class org.jgroups.protocols.pbcast.NakAckHeader
> 26: class org.jgroups.protocols.pbcast.PbcastHeader
> 27: class org.jgroups.protocols.pbcast.STABLE$StableHeader
> 28: class org.jgroups.protocols.pbcast.STATE_TRANSFER$StateHeader
> 29: class org.jgroups.protocols.SMACK$SmackHeader
> 30: class org.jgroups.Message
> 31: class org.jgroups.View
> 32: class org.jgroups.ViewId
> 33: class org.jgroups.util.List
> 34: interface org.jgroups.Address
> 35: class [Lorg.jgroups.Address;
> 36: class org.jgroups.protocols.PingRsp
> 37: class [Ljava.lang.Object;
> 38: class java.util.Vector
> 39: class org.jgroups.protocols.pbcast.JoinRsp
> 40: class org.jgroups.protocols.pbcast.Digest
> 41: class java.util.Hashtable
> 50: class org.jgroups.service.lease.LeaseRequestHeader
> 51: class org.jgroups.service.lease.LeaseResponseHeader
> 52: class org.jgroups.service.lease.DenyResponseHeader
> 53: class org.jgroups.protocols.COMPRESS$CompressHeader
> 54: class org.jgroups.protocols.FC$FcHeader
> 55: class org.jgroups.protocols.WanPipeAddress
> 16:51:49,765 DEBUG AUTOCONF:166 - frag_size=64000
> 16:51:49,843 DEBUG GMS:93 - changed role to org.jgroups.protocols.pbcast.ClientG
> msImpl
> 16:51:49,890 DEBUG UDP:359 - creating sockets and starting threads
> 16:51:49,906 INFO UDP:1048 - sockets will use interface 10.20.111.1
> 16:51:49,921 INFO UDP:1094 - socket information:
> local_addr=10.20.111.1:1172, mcast_addr=231.12.21.132:45566, bind_addr=/10.20.11
> 1.1, ttl=32
> sock: bound to 10.20.111.1:1172, receive buffer size=64000, send buffer size=320
> 00
> mcast_recv_sock: bound to 10.20.111.1:45566, send buffer size=150000, receive bu
> ffer size=80000
> mcast_send_sock: bound to 10.20.111.1:1173, send buffer size=150000, receive buf
> fer size=80000
> -------------------------------------------------------
> GMS: address is 10.20.111.1:1172
> -------------------------------------------------------
> 16:51:50,015 DEBUG UDP:1347 - created unicast receiver thread
> 16:51:50,062 DEBUG FD_SOCK:872 - waiting for client connections on 0.0.0.0/0.0.0
> .0:1174
> 16:51:50,062 DEBUG PingWaiter:107 - waiting for initial members: time_to_wait=20
> 00, got 0 rsps
> 16:51:50,062 DEBUG PingSender:57 - sending GET_MBRS_REQ
> 16:51:50,125 DEBUG UDP:803 - sending msg to 231.12.21.132:45566 (src=10.20.111.1
> :1172), headers are {PING=[PING: type=GET_MBRS_REQ, arg=null], UDP=[UDP:channel_
> name=OSCacheBus]}
> 16:51:50,156 DEBUG UDP:820 - looped back local message [dst: 231.12.21.132:45566
> , src: 10.20.111.1:1172 (2 headers), size = 0 bytes]
> 16:51:50,187 DEBUG PING:183 - discarded my own discovery request
> 16:51:50,187 DEBUG UDP:247 - received (mcast) 51 bytes from /10.20.111.1:1173 (s
> ize=51 bytes)
> 16:51:50,187 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.20.111.1:1117
> 16:51:50,218 DEBUG UDP:748 - discarded own loopback multicast packet
> 16:51:50,234 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.20.111.1
> :1117 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
> s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.20.111.1:1117, coord_addr=10
> .20.111.1:1117, is_server=false]]}
> 16:51:50,281 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.20.111.3:65180
> 16:51:50,281 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.20.111.1:1
> 117, coord_addr=10.20.111.1:1117, is_server=false]
> 16:51:50,296 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.20.111.3
> :65180 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheB
> us], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.20.111.3:65180, coord_addr=
> 10.40.10.52:4775, is_server=true]]}
> 16:51:50,312 DEBUG PingWaiter:107 - waiting for initial members: time_to_wait=17
> 50, got 1 rsps
> 16:51:50,359 DEBUG UDP:247 - received (mcast) 420 bytes from /10.40.124.4:1765 (
> size=420 bytes)
> 16:51:50,359 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.20.111.3:6
> 5180, coord_addr=10.40.10.52:4775, is_server=true]
> 16:51:50,359 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.20.111.2:1620
> 16:51:50,406 DEBUG UDP:757 - message is [dst: 231.12.21.132:45566, src: 10.40.12
> 4.4:1764 (2 headers), size = 355 bytes], headers are {NAKACK=[NAKACK: MSG, seqno
> =537, range=null], UDP=[UDP:channel_name=OSCacheBus]}
> 16:51:50,421 DEBUG PingWaiter:107 - waiting for initial members: time_to_wait=16
> 41, got 2 rsps
> 16:51:50,437 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.20.111.2
> :1620 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
> s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.20.111.2:1620, coord_addr=10
> .20.110.1:2715, is_server=true]]}
> 16:51:50,484 DEBUG UDP:247 - received (mcast) 420 bytes from /10.40.124.4:1765 (
> size=420 bytes)
> 16:51:50,484 DEBUG NAKACK:340 - message was discarded (not yet server)
> 16:51:50,546 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.20.110.3:45282
> 16:51:50,546 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.20.111.2:1
> 620, coord_addr=10.20.110.1:2715, is_server=true]
> 16:51:50,562 DEBUG UDP:757 - message is [dst: 231.12.21.132:45566, src: 10.40.12
> 4.4:1764 (2 headers), size = 355 bytes], headers are {NAKACK=[NAKACK: MSG, seqno
> =538, range=null], UDP=[UDP:channel_name=OSCacheBus]}
> 16:51:50,593 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.20.110.3
> :45282 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheB
> us], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.20.110.3:45282, coord_addr=
> 10.40.10.52:4775, is_server=true]]}
> 16:51:50,609 DEBUG PingWaiter:122 - initial mbrs are [[own_addr=10.20.111.1:1117
> , coord_addr=10.20.111.1:1117, is_server=false], [own_addr=10.20.111.3:65180, co
> ord_addr=10.40.10.52:4775, is_server=true], [own_addr=10.20.111.2:1620, coord_ad
> dr=10.20.110.1:2715, is_server=true]]
> 16:51:50,640 DEBUG NAKACK:340 - message was discarded (not yet server)
> 16:51:50,687 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.20.110.2:56053
> 16:51:50,687 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.20.110.3:4
> 5282, coord_addr=10.40.10.52:4775, is_server=true]
> 16:51:50,750 DEBUG ClientGmsImpl:66 - initial_mbrs are [[own_addr=10.20.111.1:11
> 17, coord_addr=10.20.111.1:1117, is_server=false], [own_addr=10.20.111.3:65180,
> coord_addr=10.40.10.52:4775, is_server=true], [own_addr=10.20.111.2:1620, coord_
> addr=10.20.110.1:2715, is_server=true]]
> 16:51:50,765 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.20.110.2
> :56053 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheB
> us], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.20.110.2:56053, coord_addr=
> 10.20.110.1:2715, is_server=true]]}
> 16:51:50,828 WARN ClientGmsImpl:331 - there was more than 1 candidate for coord
> inator: {10.40.10.52:4775=1, 10.20.110.1:2715=1}
> 16:51:50,875 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.10.52:4775
> 16:51:50,875 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.20.110.2:5
> 6053, coord_addr=10.20.110.1:2715, is_server=true]
> 16:51:50,890 DEBUG ClientGmsImpl:113 - sending handleJoin(10.20.111.1:1172) to 1
> 0.40.10.52:4775
> 16:51:50,906 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.10.52
> :4775 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
> s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.10.52:4775, coord_addr=10
> .20.110.1:2715, is_server=true]]}
> 16:51:50,953 DEBUG UNICAST:222 - [10.20.111.1:1172] --> DATA(10.40.10.52:4775: #
> 80, first=true)
> 16:51:51,000 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.10.52:4
> 775, coord_addr=10.20.110.1:2715, is_server=true]
> 16:51:51,000 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.10.51:3426
> 16:51:51,015 DEBUG STABLE:324 - stable task started; num_gossip_runs=3, max_goss
> ip_runs=3
> 16:51:51,046 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.10.51
> :3426 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
> s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.10.51:3426, coord_addr=10
> .20.110.1:2715, is_server=true]]}
> 16:51:51,062 DEBUG UDP:803 - sending msg to 10.40.10.52:4775 (src=10.20.111.1:11
> 72), headers are {UNICAST=[UNICAST: DATA, seqno=80], GMS=GmsHeader[JOIN_REQ]: mb
> r=10.20.111.1:1172, UDP=[UDP:channel_name=OSCacheBus]}
> 16:51:51,109 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.126.2:4228
> 16:51:51,109 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.10.51:3
> 426, coord_addr=10.20.110.1:2715, is_server=true]
> 16:51:51,156 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.126.2
> :4228 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
> s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.126.2:4228, coord_addr=10
> .20.110.1:2715, is_server=true]]}
> 16:51:51,234 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.120.3:4838
> 16:51:51,234 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.126.2:4
> 228, coord_addr=10.20.110.1:2715, is_server=true]
> 16:51:51,234 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.120.3
> :4838 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
> s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.120.3:4838, coord_addr=10
> .20.110.1:2715, is_server=true]]}
> 16:51:51,296 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.126.1:58260
> 16:51:51,296 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.120.3:4
> 838, coord_addr=10.20.110.1:2715, is_server=true]
> 16:51:51,312 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.126.1
> :58260 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheB
> us], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.126.1:58260, coord_addr=
> 10.20.110.1:2715, is_server=true]]}
> 16:51:51,390 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.120.4:3696
> 16:51:51,390 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.126.1:5
> 8260, coord_addr=10.20.110.1:2715, is_server=true]
> 16:51:51,390 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.120.4
> :3696 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
> s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.120.4:3696, coord_addr=10
> .20.110.1:2715, is_server=true]]}
> 16:51:51,453 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.121.1:3810
> 16:51:51,453 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.120.4:3
> 696, coord_addr=10.20.110.1:2715, is_server=true]
> 16:51:51,468 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.121.1
> :3810 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
> s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.121.1:3810, coord_addr=10
> .20.110.1:2715, is_server=true]]}
> 16:51:51,531 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.121.1:3
> 810, coord_addr=10.20.110.1:2715, is_server=true]
> 16:51:51,531 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.121.3:4249
> 16:51:51,578 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.121.3
> :4249 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
> s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.121.3:4249, coord_addr=10
> .20.110.1:2715, is_server=true]]}
> 16:51:51,609 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.124.1:64677
> 16:51:51,609 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.121.3:4
> 249, coord_addr=10.20.110.1:2715, is_server=true]
> 16:51:51,625 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.124.1
> :64677 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheB
> us], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.124.1:64677, coord_addr=
> 10.40.124.1:64677, is_server=false]]}
> 16:51:51,687 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.125.1:3612
> 16:51:51,687 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.124.1:6
> 4677, coord_addr=10.40.124.1:64677, is_server=false]
> 16:51:51,703 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.125.1
> :3612 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
> s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.125.1:3612, coord_addr=10
> .20.110.1:2715, is_server=true]]}
> 16:51:51,765 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.125.1:3
> 612, coord_addr=10.20.110.1:2715, is_server=true]
> 16:51:51,765 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.124.4:1764
> 16:51:51,796 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.124.4
> :1764 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
> s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.124.4:1764, coord_addr=10
> .20.110.1:2715, is_server=true]]}
> 16:51:51,843 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.126.3:4204
> 16:51:51,843 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.124.4:1
> 764, coord_addr=10.20.110.1:2715, is_server=true]
> 16:51:51,859 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.126.3
> :4204 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
> s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.126.3:4204, coord_addr=10
> .20.110.1:2715, is_server=true]]}
> 16:51:51,921 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.121.2:1403
> 16:51:51,921 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.126.3:4
> 204, coord_addr=10.20.110.1:2715, is_server=true]
> 16:51:51,937 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.121.2
> :1403 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
> s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.121.2:1403, coord_addr=10
> .20.110.1:2715, is_server=true]]}
> 16:51:52,000 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.121.2:1
> 403, coord_addr=10.20.110.1:2715, is_server=true]
> 16:51:52,000 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.121.4:1832
> 16:51:52,031 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.121.4
> :1832 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
> s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.121.4:1832, coord_addr=10
> .20.110.1:2715, is_server=true]]}
> 16:51:52,062 DEBUG UDP:247 - received (mcast) 1098 bytes from /10.40.125.2:3378
> (size=1098 bytes)
> 16:51:52,078 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.120.2:2013
> 16:51:52,078 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.121.4:1
> 832, coord_addr=10.20.110.1:2715, is_server=true]
> 16:51:52,093 DEBUG UDP:757 - message is [dst: 231.12.21.132:45566, src: 10.40.12
> 5.2:3377 (3 headers), size = 0 bytes], headers are {STABLE=[STABLE_GOSSIP]: dige
> st is [10.20.110.1:2715: [0 : 595 (595)], 10.20.110.2:56053: [1 : 1490 (1490)],
> 10.20.110.3:45282: [0 : 630 (630)], 10.20.111.1:53038: [0 : 21 (21)], 10.20.111.
> 2:1620: [0 : 952 (952)], 10.20.111.3:65180: [0 : 379 (379)], 10.40.10.51:3426: [
> 5 : 3906 (3906)], 10.40.10.52:4775: [16 : 3457 (3457)], 10.40.120.2:2013: [0 : 6
> 78 (678)], 10.40.120.3:4838: [0 : 674 (674)], 10.40.120.4:3696: [0 : 658 (658)],
> 10.40.121.1:3810: [0 : 884 (884)], 10.40.121.2:1403: [0 : 631 (631)], 10.40.121
> .3:4249: [0 : 598 (598)], 10.40.121.4:1832: [0 : 773 (773)], 10.40.124.1:55644:
> [0 : 512 (512)], 10.40.124.2:31634: [0 : 652 (652)], 10.40.124.3:15988: [0 : 790
> (790)], 10.40.124.4:1764: [0 : 538 (538)], 10.40.125.1:3612: [0 : 804 (804)], 1
> 0.40.125.2:3377: [0 : 539 (539)], 10.40.125.3:1739: [0 : 774 (774)], 10.40.125.4
> :2799: [0 : 756 (756)], 10.40.126.1:58260: [0 : 52 (52)], 10.40.126.2:4228: [0 :
> 50 (50)], 10.40.126.3:4204: [0 : 51 (51)], 10.40.126.4:21141: [0 : 643 (643)]],
> NAKACK=[NAKACK: MSG, seqno=540, range=null], UDP=[UDP:channel_name=OSCacheBus]}
> 16:51:52,109 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.120.2
> :2013 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
> s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.120.2:2013, coord_addr=10
> .20.110.1:2715, is_server=true]]}
> 16:51:52,296 DEBUG NAKACK:340 - message was discarded (not yet server)
> 16:51:52,343 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.120.2:2
> 013, coord_addr=10.20.110.1:2715, is_server=true]
> 16:51:52,343 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.124.3:15988
> 16:51:52,390 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.124.3
> :15988 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheB
> us], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.124.3:15988, coord_addr=
> 10.20.110.1:2715, is_server=true]]}
> 16:51:52,437 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.124.3:1
> 5988, coord_addr=10.20.110.1:2715, is_server=true]
> 16:51:52,437 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.125.4:2799
> 16:51:52,468 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.125.4
> :2799 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
> s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.125.4:2799, coord_addr=10
> .20.110.1:2715, is_server=true]]}
> 16:51:52,515 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.125.2:3377
> 16:51:52,515 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.125.4:2
> 799, coord_addr=10.20.110.1:2715, is_server=true]
> 16:51:52,531 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.125.2
> :3377 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
> s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.125.2:3377, coord_addr=10
> .20.110.1:2715, is_server=true]]}
> 16:51:52,593 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.124.2:31634
> 16:51:52,593 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.125.2:3
> 377, coord_addr=10.20.110.1:2715, is_server=true]
> 16:51:52,609 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.124.2
> :31634 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheB
> us], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.124.2:31634, coord_addr=
> 10.20.110.1:2715, is_server=true]]}
> 16:51:52,671 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.126.4:21141
> 16:51:52,671 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.124.2:3
> 1634, coord_addr=10.20.110.1:2715, is_server=true]
> 16:51:52,687 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.126.4
> :21141 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheB
> us], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.126.4:21141, coord_addr=
> 10.20.110.1:2715, is_server=true]]}
> 16:51:52,750 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.124.1:64720
> 16:51:52,750 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.126.4:2
> 1141, coord_addr=10.20.110.1:2715, is_server=true]
> 16:51:52,765 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.124.1
> :64720 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheB
> us], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.124.1:64720, coord_addr=
> 10.40.124.1:64720, is_server=false]]}
> 16:51:52,828 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.124.1:6
> 4720, coord_addr=10.40.124.1:64720, is_server=false]
> 16:51:52,828 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.125.3:1739
> 16:51:52,875 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.125.3
> :1739 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
> s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.125.3:1739, coord_addr=10
> .20.110.1:2715, is_server=true]]}
> 16:51:52,906 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.125.3:1
> 739, coord_addr=10.20.110.1:2715, is_server=true]
> 16:51:52,906 DEBUG UDP:1542 - received (ucast) 62 bytes from /10.40.10.52:4775
> 16:51:52,953 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.10.52
> :4775 (2 headers), size = 0 bytes], headers are {UNICAST=[UNICAST: DATA_ACK, seq
> no=80], UDP=[UDP:channel_name=OSCacheBus]}
> 16:51:52,984 DEBUG UDP:1542 - received (ucast) 82 bytes from /10.40.10.52:4775
> 16:51:52,984 DEBUG UNICAST:387 - [10.20.111.1:1172] <-- ACK(10.40.10.52:4775: #8
> 0)
> 16:51:53,000 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.10.52
> :4775 (3 headers), size = 0 bytes], headers are {UNICAST=[UNICAST: DATA, seqno=6
> 1], GMS=GmsHeader[JOIN_RSP]: join_rsp=null, UDP=[UDP:channel_name=OSCacheBus]}
> 16:51:53,046 DEBUG UNICAST:402 - [10.20.111.1:1172] --> ACK(10.40.10.52:4775: #6
> 1)
> 16:51:53,078 DEBUG UNICAST:344 - [10.20.111.1:1172] <-- DATA(10.40.10.52:4775: #
> 61, first=true
> 16:51:53,078 DEBUG UDP:803 - sending msg to 10.40.10.52:4775 (src=10.20.111.1:11
> 72), headers are {UNICAST=[UNICAST: DATA_ACK, seqno=61], UDP=[UDP:channel_name=O
> SCacheBus]}
> 16:51:53,093 WARN ClientGmsImpl:118 - join(10.20.111.1:1172) failed, retrying
> 16:51:54,812 DEBUG UDP:247 - received (mcast) 51 bytes from /10.40.124.1:64678 (
> size=51 bytes)
> 16:51:54,828 DEBUG UDP:757 - message is [dst: 231.12.21.132:45566, src: 10.40.12
> 4.1:64677 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCac
> heBus], PING=[PING: type=GET_MBRS_REQ, arg=null]}
> 16:51:54,859 DEBUG PING:195 - received GET_MBRS_REQ from 10.40.124.1:64677, send
> ing response [PING: type=GET_MBRS_RSP, arg=[own_addr=10.20.111.1:1172, coord_add
> r=10.20.111.1:1172, is_server=false]]
> 16:51:54,890 DEBUG UDP:247 - received (mcast) 51 bytes from /10.20.111.1:1118 (s
> ize=51 bytes)
> 16:51:54,890 DEBUG UDP:803 - sending msg to 10.40.124.1:64677 (src=10.20.111.1:1
> 172), headers are {PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.20.111.1:1172
> , coord_addr=10.20.111.1:1172, is_server=false]], UDP=[UDP:channel_name=OSCacheB
> us]}
> 16:51:54,921 DEBUG UDP:757 - message is [dst: 231.12.21.132:45566, src: 10.20.11
> 1.1:1117 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCach
> eBus], PING=[PING: type=GET_MBRS_REQ, arg=null]}
> 16:51:55,000 DEBUG UDP:247 - received (mcast) 420 bytes from /10.40.121.2:1404 (
> size=420 bytes)
> 16:51:55,000 DEBUG PING:195 - received GET_MBRS_REQ from 10.20.111.1:1117, sendi
> ng response [PING: type=GET_MBRS_RSP, arg=[own_addr=10.20.111.1:1172, coord_addr
> =10.20.111.1:1172, is_server=false]]
> 16:51:55,015 DEBUG UDP:757 - message is [dst: 231.12.21.132:45566, src: 10.40.12
> 1.2:1403 (2 headers), size = 355 bytes], headers are {NAKACK=[NAKACK: MSG, seqno
> =632, range=null], UDP=[UDP:channel_name=OSCacheBus]}
> 16:51:55,046 DEBUG UDP:803 - sending msg to 10.20.111.1:1117 (src=10.20.111.1:11
> 72), headers are {PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.20.111.1:1172,
> coord_addr=10.20.111.1:1172, is_server=false]], UDP=[UDP:channel_name=OSCacheBu
> s]}
> 16:51:55,078 DEBUG UDP:247 - received (mcast) 420 bytes from /10.40.121.2:1404 (
> size=420 bytes)
> 16:51:55,078 DEBUG NAKACK:340 - message was discarded (not yet server)
> 16:51:55,140 DEBUG UDP:757 - message is [dst: 231.12.21.132:45566, src: 10.40.12
> 1.2:1403 (2 headers), size = 355 bytes], headers are {NAKACK=[NAKACK: MSG, seqno
> =633, range=null], UDP=[UDP:channel_name=OSCacheBus]}
> 16:51:55,156 DEBUG PingWaiter:95 - clearing old responses: [[own_addr=10.20.111.
> 1:1117, coord_addr=10.20.111.1:1117, is_server=false], [own_addr=10.20.111.3:651
> 80, coord_addr=10.40.10.52:4775, is_server=true], [own_addr=10.20.111.2:1620, co
> ord_addr=10.20.110.1:2715, is_server=true], [own_addr=10.20.110.3:45282, coord_a
> ddr=10.40.10.52:4775, is_server=true], [own_addr=10.20.110.2:56053, coord_addr=1
> 0.20.110.1:2715, is_server=true], [own_addr=10.40.10.52:4775, coord_addr=10.20.1
> 10.1:2715, is_server=true], [own_addr=10.40.10.51:3426, coord_addr=10.20.110.1:2
> 715, is_server=true], [own_addr=10.40.126.2:4228, coord_addr=10.20.110.1:2715, i
> s_server=true], [own_addr=10.40.120.3:4838, coord_addr=10.20.110.1:2715, is_serv
> er=true], [own_addr=10.40.126.1:58260, coord_addr=10.20.110.1:2715, is_server=tr
> ue], [own_addr=10.40.120.4:3696, coord_addr=10.20.110.1:2715, is_server=true], [
> own_addr=10.40.121.1:3810, coord_addr=10.20.110.1:2715, is_server=true], [own_ad
> dr=10.40.121.3:4249, coord_addr=10.20.110.1:2715, is_server=true], [own_addr=10.
> 40.124.1:64677, coord_addr=10.40.124.1:64677, is_server=false], [own_addr=10.40.
> 125.1:3612, coord_addr=10.20.110.1:2715, is_server=true], [own_addr=10.40.124.4:
> 1764, coord_addr=10.20.110.1:2715, is_server=true], [own_addr=10.40.126.3:4204,
> coord_addr=10.20.110.1:2715, is_server=true], [own_addr=10.40.121.2:1403, coord_
> addr=10.20.110.1:2715, is_server=true], [own_addr=10.40.121.4:1832, coord_addr=1
> 0.20.110.1:2715, is_server=true], [own_addr=10.40.120.2:2013, coord_addr=10.20.1
> 10.1:2715, is_server=true], [own_addr=10.40.124.3:15988, coord_addr=10.20.110.1:
> 2715, is_server=true], [own_addr=10.40.125.4:2799, coord_addr=10.20.110.1:2715,
> is_server=true], [own_addr=10.40.125.2:3377, coord_addr=10.20.110.1:2715, is_ser
> ver=true], [own_addr=10.40.124.2:31634, coord_addr=10.20.110.1:2715, is_server=t
> rue], [own_addr=10.40.126.4:21141, coord_addr=10.20.110.1:2715, is_server=true],
> [own_addr=10.40.124.1:64720, coord_addr=10.40.124.1:64720, is_server=false], [o
> wn_addr=10.40.125.3:1739, coord_addr=10.20.110.1:2715, is_server=true]]
> 16:51:55,187 DEBUG UDP:247 - received (mcast) 51 bytes from /10.40.124.1:64721 (
> size=51 bytes)
> 16:51:55,187 DEBUG NAKACK:340 - message was discarded (not yet server)
> 16:51:55,468 DEBUG PingWaiter:107 - waiting for initial members: time_to_wait=20
> 00, got 0 rsps
> 16:51:55,484 DEBUG PingSender:57 - sending GET_MBRS_REQ
> 16:51:55,500 DEBUG UDP:757 - message is [dst: 231.12.21.132:45566, src: 10.40.12
> 4.1:64720 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCac
> heBus], PING=[PING: type=GET_MBRS_REQ, arg=null]}
> 16:51:55,546 DEBUG UDP:803 - sending msg to 231.12.21.132:45566 (src=10.20.111.1
> :1172), headers are {PING=[PING: type=GET_MBRS_REQ, arg=null], UDP=[UDP:channel_
> name=OSCacheBus]}
> 16:51:55,578 DEBUG PING:195 - received GET_MBRS_REQ from 10.40.124.1:64720, send
> ing response [PING: type=GET_MBRS_RSP, arg=[own_addr=10.20.111.1:1172, coord_add
> r=10.20.111.1:1172, is_server=false]]
> 16:51:55,609 DEBUG UDP:820 - looped back local message [dst: 231.12.21.132:45566
> , src: 10.20.111.1:1172 (2 headers), size = 0 bytes]
> 16:51:55,671 DEBUG PING:183 - discarded my own discovery request
> 16:51:55,671 DEBUG UDP:247 - received (mcast) 51 bytes from /10.20.111.1:1173 (s
> ize=51 bytes)
> 16:51:55,671 DEBUG UDP:803 - sending msg to 10.40.124.1:64720 (src=10.20.111.1:1
> 172), headers are {PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.20.111.1:1172
> , coord_addr=10.20.111.1:1172, is_server=false]], UDP=[UDP:channel_name=OSCacheB
> us]}
> 16:51:55,671 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.20.111.1:1117
> 16:51:55,703 DEBUG UDP:748 - discarded own loopback multicast packet
> 16:51:55,750 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.20.111.1
> :1117 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
> s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.20.111.1:1117, coord_addr=10
> .20.111.1:1117, is_server=false]]}
> 16:51:55,812 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.20.111.1:1
> 117, coord_addr=10.20.111.1:1117, is_server=false]
> 16:51:55,812 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.20.111.3:65180
> 16:51:55,828 DEBUG PingWaiter:107 - waiting for initial members: time_to_wait=16
> 40, got 1 rsps
> 16:51:55,843 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.20.111.3
> :65180 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheB
> us], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.20.111.3:65180, coord_addr=
> 10.40.10.52:4775, is_server=true]]}
> 16:51:55,906 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.20.111.3:6
> 5180, coord_addr=10.40.10.52:4775, is_server=true]
> 16:51:55,906 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.20.110.2:56053
> 16:51:55,937 DEBUG PingWaiter:107 - waiting for initial members: time_to_wait=15
> 31, got 2 rsps
> 16:51:55,937 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.20.110.2
> :56053 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheB
> us], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.20.110.2:56053, coord_addr=
> 10.20.110.1:2715, is_server=true]]}
> 16:51:56,015 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.20.110.3:45282
> 16:51:56,015 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.20.110.2:5
> 6053, coord_addr=10.20.110.1:2715, is_server=true]
> 16:51:56,015 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.20.110.3
> :45282 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheB
> us], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.20.110.3:45282, coord_addr=
> 10.40.10.52:4775, is_server=true]]}
> 16:51:56,046 DEBUG PingWaiter:122 - initial mbrs are [[own_addr=10.20.111.1:1117
> , coord_addr=10.20.111.1:1117, is_server=false], [own_addr=10.20.111.3:65180, co
> ord_addr=10.40.10.52:4775, is_server=true], [own_addr=10.20.110.2:56053, coord_a
> ddr=10.20.110.1:2715, is_server=true]]
> 16:51:56,078 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.20.111.2:1620
> 16:51:56,078 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.20.110.3:4
> 5282, coord_addr=10.40.10.52:4775, is_server=true]
> 16:51:56,125 DEBUG ClientGmsImpl:66 - initial_mbrs are [[own_addr=10.20.111.1:11
> 17, coord_addr=10.20.111.1:1117, is_server=false], [own_addr=10.20.111.3:65180,
> coord_addr=10.40.10.52:4775, is_server=true], [own_addr=10.20.110.2:56053, coord
> _addr=10.20.110.1:2715, is_server=true]]
> 16:51:56,140 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.20.111.2
> :1620 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
> s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.20.111.2:1620, coord_addr=10
> .20.110.1:2715, is_server=true]]}
> 16:51:56,203 WARN ClientGmsImpl:331 - there was more than 1 candidate for coord
> inator: {10.40.10.52:4775=1, 10.20.110.1:2715=1}
> 16:51:56,250 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.10.51:3426
> 16:51:56,250 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.20.111.2:1
> 620, coord_addr=10.20.110.1:2715, is_server=true]
> 16:51:56,281 DEBUG ClientGmsImpl:113 - sending handleJoin(10.20.111.1:1172) to 1
> 0.40.10.52:4775
> 16:51:56,281 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.10.51
> :3426 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
> s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.10.51:3426, coord_addr=10
> .20.110.1:2715, is_server=true]]}
> 16:51:56,328 DEBUG UNICAST:222 - [10.20.111.1:1172] --> DATA(10.40.10.52:4775: #
> 81, first=false)
> 16:51:56,375 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.124.1:64677
> 16:51:56,375 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.10.51:3
> 426, coord_addr=10.20.110.1:2715, is_server=true]
> 16:51:56,406 DEBUG UDP:803 - sending msg to 10.40.10.52:4775 (src=10.20.111.1:11
> 72), headers are {UNICAST=[UNICAST: DATA, seqno=81], GMS=GmsHeader[JOIN_REQ]: mb
> r=10.20.111.1:1172, UDP=[UDP:channel_name=OSCacheBus]}
> 16:51:56,406 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.124.1
> :64677 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheB
> us], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.124.1:64677, coord_addr=
> 10.40.124.1:64677, is_server=false]]}
> 16:51:56,515 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.121.1:3810
> 16:51:56,515 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.124.1:6
> 4677, coord_addr=10.40.124.1:64677, is_server=false]
> 16:51:56,515 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.121.1
> :3810 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
> s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.121.1:3810, coord_addr=10
> .20.110.1:2715, is_server=true]]}
> 16:51:56,593 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.121.1:3
> 810, coord_addr=10.20.110.1:2715, is_server=true]
> 16:51:56,593 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.10.52:4775
> 16:51:56,625 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.10.52
> :4775 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
> s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.10.52:4775, coord_addr=10
> .20.110.1:2715, is_server=true]]}
> 16:51:56,671 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.10.52:4
> 775, coord_addr=10.20.110.1:2715, is_server=true]
> 16:51:56,671 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.124.3:15988
> 16:51:56,703 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.124.3
> :15988 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheB
> us], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.124.3:15988, coord_addr=
> 10.20.110.1:2715, is_server=true]]}
> 16:51:56,750 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.126.2:4228
> 16:51:56,750 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.124.3:1
> 5988, coord_addr=10.20.110.1:2715, is_server=true]
> 16:51:56,765 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.126.2
> :4228 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
> s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.126.2:4228, coord_addr=10
> .20.110.1:2715, is_server=true]]}
> 16:51:56,765 DEBUG UDP:247 - received (mcast) 420 bytes from /10.40.124.4:1765 (
> size=420 bytes)
> 16:51:56,828 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.120.3:4838
> 16:51:56,828 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.126.2:4
> 228, coord_addr=10.20.110.1:2715, is_server=true]
> 16:51:56,843 DEBUG UDP:757 - message is [dst: 231.12.21.132:45566, src: 10.40.12
> 4.4:1764 (2 headers), size = 355 bytes], headers are {NAKACK=[NAKACK: MSG, seqno
> =539, range=null], UDP=[UDP:channel_name=OSCacheBus]}
> 16:51:56,859 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.120.3
> :4838 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
> s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.120.3:4838, coord_addr=10
> .20.110.1:2715, is_server=true]]}
> 16:51:56,921 DEBUG UDP:247 - received (mcast) 420 bytes from /10.40.124.4:1765 (
> size=420 bytes)
> 16:51:56,921 DEBUG NAKACK:340 - message was discarded (not yet server)
> 16:51:56,953 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.125.4:2799
> 16:51:56,953 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.120.3:4
> 838, coord_addr=10.20.110.1:2715, is_server=true]
> 16:51:56,984 DEBUG UDP:757 - message is [dst: 231.12.21.132:45566, src: 10.40.12
> 4.4:1764 (2 headers), size = 355 bytes], headers are {NAKACK=[NAKACK: MSG, seqno
> =540, range=null], UDP=[UDP:channel_name=OSCacheBus]}
> 16:51:57,000 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.125.4
> :2799 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
> s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.125.4:2799, coord_addr=10
> .20.110.1:2715, is_server=true]]}
> 16:51:57,062 DEBUG NAKACK:340 - message was discarded (not yet server)
> 16:51:57,109 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.126.1:58260
> 16:51:57,109 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.125.4:2
> 799, coord_addr=10.20.110.1:2715, is_server=true]
> 16:51:57,125 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.126.1
> :58260 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheB
> us], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.126.1:58260, coord_addr=
> 10.20.110.1:2715, is_server=true]]}
> 16:51:57,203 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.126.1:5
> 8260, coord_addr=10.20.110.1:2715, is_server=true]
> 16:51:57,203 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.121.4:1832
> 16:51:57,234 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.121.4
> :1832 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
> s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.121.4:1832, coord_addr=10
> .20.110.1:2715, is_server=true]]}
> 16:51:57,281 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.121.2:1403
> 16:51:57,281 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.121.4:1
> 832, coord_addr=10.20.110.1:2715, is_server=true]
> 16:51:57,296 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.121.2
> :1403 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
> s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.121.2:1403, coord_addr=10
> .20.110.1:2715, is_server=true]]}
> 16:51:57,359 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.121.2:1
> 403, coord_addr=10.20.110.1:2715, is_server=true]
> 16:51:57,359 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.126.3:4204
> 16:51:57,390 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.126.3
> :4204 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
> s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.126.3:4204, coord_addr=10
> .20.110.1:2715, is_server=true]]}
> 16:51:57,437 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.124.4:1764
> 16:51:57,437 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.126.3:4
> 204, coord_addr=10.20.110.1:2715, is_server=true]
> 16:51:57,453 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.124.4
> :1764 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
> s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.124.4:1764, coord_addr=10
> .20.110.1:2715, is_server=true]]}
> 16:51:57,515 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.125.3:1739
> 16:51:57,515 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.124.4:1
> 764, coord_addr=10.20.110.1:2715, is_server=true]
> 16:51:57,531 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.125.3
> :1739 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
> s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.125.3:1739, coord_addr=10
> .20.110.1:2715, is_server=true]]}
> 16:51:57,593 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.125.3:1
> 739, coord_addr=10.20.110.1:2715, is_server=true]
> 16:51:57,593 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.121.3:4249
> 16:51:57,625 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.121.3
> :4249 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
> s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.121.3:4249, coord_addr=10
> .20.110.1:2715, is_server=true]]}
> 16:51:57,671 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.121.3:4
> 249, coord_addr=10.20.110.1:2715, is_server=true]
> 16:51:57,671 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.124.2:31634
> 16:51:57,703 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.124.2
> :31634 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheB
> us], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.124.2:31634, coord_addr=
> 10.20.110.1:2715, is_server=true]]}
> 16:51:57,750 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.124.2:3
> 1634, coord_addr=10.20.110.1:2715, is_server=true]
> 16:51:57,750 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.125.2:3377
> 16:51:57,781 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.125.2
> :3377 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
> s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.125.2:3377, coord_addr=10
> .20.110.1:2715, is_server=true]]}
> 16:51:57,828 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.120.4:3696
> 16:51:57,828 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.125.2:3
> 377, coord_addr=10.20.110.1:2715, is_server=true]
> 16:51:57,843 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.120.4
> :3696 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
> s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.120.4:3696, coord_addr=10
> .20.110.1:2715, is_server=true]]}
> 16:51:57,906 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.120.4:3
> 696, coord_addr=10.20.110.1:2715, is_server=true]
> 16:51:57,906 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.125.1:3612
> 16:51:57,937 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.125.1
> :3612 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
> s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.125.1:3612, coord_addr=10
> .20.110.1:2715, is_server=true]]}
> 16:51:57,984 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.120.2:2013
> 16:51:57,984 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.125.1:3
> 612, coord_addr=10.20.110.1:2715, is_server=true]
> 16:51:58,000 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.120.2
> :2013 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
> s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.120.2:2013, coord_addr=10
> .20.110.1:2715, is_server=true]]}
> 16:51:58,046 DEBUG UDP:247 - received (mcast) 420 bytes from /10.40.124.4:1765 (
> size=420 bytes)
> 16:51:58,062 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.126.4:21141
> 16:51:58,062 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.120.2:2
> 013, coord_addr=10.20.110.1:2715, is_server=true]
> 16:51:58,078 DEBUG UDP:757 - message is [dst: 231.12.21.132:45566, src: 10.40.12
> 4.4:1764 (2 headers), size = 355 bytes], headers are {NAKACK=[NAKACK: MSG, seqno
> =541, range=null], UDP=[UDP:channel_name=OSCacheBus]}
> 16:51:58,093 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.126.4
> :21141 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheB
> us], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.126.4:21141, coord_addr=
> 10.20.110.1:2715, is_server=true]]}
> 16:51:58,156 DEBUG UDP:247 - received (mcast) 420 bytes from /10.40.124.4:1765 (
> size=420 bytes)
> 16:51:58,156 DEBUG NAKACK:340 - message was discarded (not yet server)
> 16:51:58,187 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.126.4:2
> 1141, coord_addr=10.20.110.1:2715, is_server=true]
> 16:51:58,187 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.124.1:64720
> 16:51:58,218 DEBUG UDP:757 - message is [dst: 231.12.21.132:45566, src: 10.40.12
> 4.4:1764 (2 headers), size = 355 bytes], headers are {NAKACK=[NAKACK: MSG, seqno
> =542, range=null], UDP=[UDP:channel_name=OSCacheBus]}
> 16:51:58,265 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.124.1
> :64720 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheB
> us], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.124.1:64720, coord_addr=
> 10.40.124.1:64720, is_server=false]]}
> 16:51:58,296 DEBUG NAKACK:340 - message was discarded (not yet server)
> 16:51:58,343 DEBUG UDP:1542 - received (ucast) 62 bytes from /10.40.10.52:4775
> 16:51:58,343 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.124.1:6
> 4720, coord_addr=10.40.124.1:64720, is_server=false]
> 16:51:58,375 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.10.52
> :4775 (2 headers), size = 0 bytes], headers are {UNICAST=[UNICAST: DATA_ACK, seq
> no=81], UDP=[UDP:channel_name=OSCacheBus]}
> 16:51:58,406 DEBUG UDP:247 - received (mcast) 420 bytes from /10.40.121.2:1404 (
> size=420 bytes)
> 16:51:58,421 DEBUG UDP:1542 - received (ucast) 82 bytes from /10.40.10.52:4775
> 16:51:58,421 DEBUG UNICAST:387 - [10.20.111.1:1172] <-- ACK(10.40.10.52:4775: #8
> 1)
> 16:51:58,453 DEBUG UDP:757 - message is [dst: 231.12.21.132:45566, src: 10.40.12
> 1.2:1403 (2 headers), size = 355 bytes], headers are {NAKACK=[NAKACK: MSG, seqno
> =634, range=null], UDP=[UDP:channel_name=OSCacheBus]}
> 16:51:58,453 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.10.52
> :4775 (3 headers), size = 0 bytes], headers are {UNICAST=[UNICAST: DATA, seqno=6
> 2], GMS=GmsHeader[JOIN_RSP]: join_rsp=null, UDP=[UDP:channel_name=OSCacheBus]}
> 16:51:58,515 DEBUG UDP:247 - received (mcast) 420 bytes from /10.40.121.2:1404 (
> size=420 bytes)
> 16:51:58,515 DEBUG NAKACK:340 - message was discarded (not yet server)
> 16:51:58,578 DEBUG UDP:757 - message is [dst: 231.12.21.132:45566, src: 10.40.12
> 1.2:1403 (2 headers), size = 355 bytes], headers are {NAKACK=[NAKACK: MSG, seqno
> =635, range=null], UDP=[UDP:channel_name=OSCacheBus]}
> 16:51:58,578 DEBUG UNICAST:402 - [10.20.111.1:1172] --> ACK(10.40.10.52:4775: #6
> 2)
> 16:51:58,625 DEBUG UDP:247 - received (mcast) 420 bytes from /10.40.121.3:4250 (
> size=420 bytes)
> 16:51:58,625 DEBUG NAKACK:340 - message was discarded (not yet server)
> 16:51:58,640 DEBUG UNICAST:344 - [10.20.111.1:1172] <-- DATA(10.40.10.52:4775: #
> 62, first=false
> 16:51:58,640 DEBUG UDP:803 - sending msg to 10.40.10.52:4775 (src=10.20.111.1:11
> 72), headers are {UNICAST=[UNICAST: DATA_ACK, seqno=62], UDP=[UDP:channel_name=O
> SCacheBus]}
> 16:51:58,656 DEBUG UDP:757 - message is [dst: 231.12.21.132:45566, src: 10.40.12
> 1.3:4249 (2 headers), size = 355 bytes], headers are {NAKACK=[NAKACK: MSG, seqno
> =599, range=null], UDP=[UDP:channel_name=OSCacheBus]}
> 16:51:58,703 WARN ClientGmsImpl:118 - join(10.20.111.1:1172) failed, retrying
> 16:51:58,765 DEBUG UDP:247 - received (mcast) 420 bytes from /10.40.121.3:4250 (
> size=420 bytes)
> 16:51:58,765 DEBUG NAKACK:340 - message was discarded (not yet server)
> 16:51:58,796 DEBUG UDP:757 - message is [dst: 231.12.21.132:45566, src: 10.40.12
> 1.3:4249 (2 headers), size = 355 bytes], headers are {NAKACK=[NAKACK: MSG, seqno
> =600, range=null], UDP=[UDP:channel_name=OSCacheBus]}
> 16:51:58,843 DEBUG UDP:247 - received (mcast) 420 bytes from /10.40.124.4:1765 (
> size=420 bytes)
> 16:51:58,843 DEBUG NAKACK:340 - message was discarded (not yet server)
> 16:51:58,875 DEBUG UDP:757 - message is [dst: 231.12.21.132:45566, src: 10.40.12
> 4.4:1764 (2 headers), size = 355 bytes], headers are {NAKACK=[NAKACK: MSG, seqno
> =543, range=null], UDP=[UDP:channel_name=OSCacheBus]}
> 16:51:58,906 DEBUG UDP:247 - received (mcast) 420 bytes from /10.40.124.4:1765 (
> size=420 bytes)
> 16:51:58,906 DEBUG NAKACK:340 - message was discarded (not yet server)
> 16:51:58,937 DEBUG UDP:757 - message is [dst: 231.12.21.132:45566, src: 10.40.12
> 4.4:1764 (2 headers), size = 355 bytes], headers are {NAKACK=[NAKACK: MSG, seqno
> =544, range=null], UDP=[UDP:channel_name=OSCacheBus]}
> 16:51:58,984 DEBUG NAKACK:340 - message was discarded (not yet server)
> 16:52:00,734 DEBUG UDP:247 - received (mcast) 420 bytes from /10.40.125.1:3613 (
> size=420 bytes)
> 16:52:00,750 DEBUG UDP:757 - message is [dst: 231.12.21.132:45566, src: 10.40.12
> 5.1:3612 (2 headers), size = 355 bytes], headers are {NAKACK=[NAKACK: MSG, seqno
> =805, range=null], UDP=[UDP:channel_name=OSCacheBus]}
> 16:52:00,781 DEBUG UDP:247 - received (mcast) 420 bytes from /10.40.125.1:3613 (
> size=420 bytes)
> 16:52:00,781 DEBUG NAKACK:340 - message was discarded (not yet server)
> 16:52:00,796 DEBUG PingWaiter:95 - clearing old responses: [[own_addr=10.20.111.
> 1:1117, coord_addr=10.20.111.1:1117, is_server=false], [own_addr=10.20.111.3:651
> 80, coord_addr=10.40.10.52:4775, is_server=true], [own_addr=10.20.110.2:56053, c
> oord_addr=10.20.110.1:2715, is_server=true], [own_addr=10.20.110.3:45282, coord_
> addr=10.40.10.52:4775, is_server=true], [own_addr=10.20.111.2:1620, coord_addr=1
> 0.20.110.1:2715, is_server=true], [own_addr=10.40.10.51:3426, coord_addr=10.20.1
> 10.1:2715, is_server=true], [own_addr=10.40.124.1:64677, coord_addr=10.40.124.1:
> 64677, is_server=false], [own_addr=10.40.121.1:3810, coord_addr=10.20.110.1:2715
> , is_server=true], [own_addr=10.40.10.52:4775, coord_addr=10.20.110.1:2715, is_s
> erver=true], [own_addr=10.40.124.3:15988, coord_addr=10.20.110.1:2715, is_server
> =true], [own_addr=10.40.126.2:4228, coord_addr=10.20.110.1:2715, is_server=true]
> , [own_addr=10.40.120.3:4838, coord_addr=10.20.110.1:2715, is_server=true], [own
> _addr=10.40.125.4:2799, coord_addr=10.20.110.1:2715, is_server=true], [own_addr=
> 10.40.126.1:58260, coord_addr=10.20.110.1:2715, is_server=true], [own_addr=10.40
> .121.4:1832, coord_addr=10.20.110.1:2715, is_server=true], [own_addr=10.40.121.2
> :1403, coord_addr=10.20.110.1:2715, is_server=true], [own_addr=10.40.126.3:4204,
> coord_addr=10.20.110.1:2715, is_server=true], [own_addr=10.40.124.4:1764, coord
> _addr=10.20.110.1:2715, is_server=true], [own_addr=10.40.125.3:1739, coord_addr=
> 10.20.110.1:2715, is_server=true], [own_addr=10.40.121.3:4249, coord_addr=10.20.
> 110.1:2715, is_server=true], [own_addr=10.40.124.2:31634, coord_addr=10.20.110.1
> :2715, is_server=true], [own_addr=10.40.125.2:3377, coord_addr=10.20.110.1:2715,
> is_server=true], [own_addr=10.40.120.4:3696, coord_addr=10.20.110.1:2715, is_se
> rver=true], [own_addr=10.40.125.1:3612, coord_addr=10.20.110.1:2715, is_server=t
> rue], [own_addr=10.40.120.2:2013, coord_addr=10.20.110.1:2715, is_server=true],
> [own_addr=10.40.126.4:21141, coord_addr=10.20.110.1:2715, is_server=true], [own_
> addr=10.40.124.1:64720, coord_addr=10.40.124.1:64720, is_server=false]]
> 16:52:00,812 DEBUG UDP:757 - message is [dst: 231.12.21.132:45566, src: 10.40.12
> 5.1:3612 (2 headers), size = 355 bytes], headers are {NAKACK=[NAKACK: MSG, seqno
> =806, range=null], UDP=[UDP:channel_name=OSCacheBus]}
> 16:52:01,093 DEBUG PingWaiter:107 - waiting for initial members: time_to_wait=20
> 00, got 0 rsps
> 16:52:01,093 DEBUG PingSender:57 - sending GET_MBRS_REQ
> 16:52:01,140 DEBUG NAKACK:340 - message was discarded (not yet server)
> 16:52:01,171 DEBUG UDP:803 - sending msg to 231.12.21.132:45566 (src=10.20.111.1
> :1172), headers are {PING=[PING: type=GET_MBRS_REQ, arg=null], UDP=[UDP:channel_
> name=OSCacheBus]}
> 16:52:01,218 DEBUG UDP:820 - looped back local message [dst: 231.12.21.132:45566
> , src: 10.20.111.1:1172 (2 headers), size = 0 bytes]
> 16:52:01,234 DEBUG PING:183 - discarded my own discovery request
> 16:52:01,234 DEBUG UDP:247 - received (mcast) 51 bytes from /10.20.111.1:1173 (s
> ize=51 bytes)
> 16:52:01,234 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.20.111.1:1117
> 16:52:01,265 DEBUG UDP:748 - discarded own loopback multicast packet
> 16:52:01,281 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.20.111.1
> :1117 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
> s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.20.111.1:1117, coord_addr=10
> .20.111.1:1117, is_server=false]]}
> 16:52:01,343 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.20.111.3:65180
> 16:52:01,343 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.20.111.1:1
> 117, coord_addr=10.20.111.1:1117, is_server=false]
> 16:52:01,359 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.20.111.3
> :65180 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheB
> us], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.20.111.3:65180, coord_addr=
> 10.40.10.52:4775, is_server=true]]}
> 16:52:01,375 DEBUG PingWaiter:107 - waiting for initial members: time_to_wait=17
> 18, got 1 rsps
> 16:52:01,421 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.20.110.2:56053
> 16:52:01,421 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.20.111.3:6
> 5180, coord_addr=10.40.10.52:4775, is_server=true]
> 16:52:01,453 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.20.110.2
> :56053 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheB
> us], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.20.110.2:56053, coord_addr=
> 10.20.110.1:2715, is_server=true]]}
> 16:52:01,484 DEBUG PingWaiter:107 - waiting for initial members: time_to_wait=16
> 09, got 2 rsps
> 16:52:01,531 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.20.110.2:5
> 6053, coord_addr=10.20.110.1:2715, is_server=true]
> 16:52:01,531 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.20.110.3:45282
> 16:52:01,562 DEBUG PingWaiter:122 - initial mbrs are [[own_addr=10.20.111.1:1117
> , coord_addr=10.20.111.1:1117, is_server=false], [own_addr=10.20.111.3:65180, co
> ord_addr=10.40.10.52:4775, is_server=true], [own_addr=10.20.110.2:56053, coord_a
> ddr=10.20.110.1:2715, is_server=true]]
> 16:52:01,578 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.20.110.3
> :45282 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheB
> us], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.20.110.3:45282, coord_addr=
> 10.40.10.52:4775, is_server=true]]}
> 16:52:01,625 DEBUG ClientGmsImpl:66 - initial_mbrs are [[own_addr=10.20.111.1:11
> 17, coord_addr=10.20.111.1:1117, is_server=false], [own_addr=10.20.111.3:65180,
> coord_addr=10.40.10.52:4775, is_server=true], [own_addr=10.20.110.2:56053, coord
> _addr=10.20.110.1:2715, is_server=true]]
> 16:52:01,671 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.20.111.2:1620
> 16:52:01,671 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.20.110.3:4
> 5282, coord_addr=10.40.10.52:4775, is_server=true]
> 16:52:01,718 WARN ClientGmsImpl:331 - there was more than 1 candidate for coord
> inator: {10.40.10.52:4775=1, 10.20.110.1:2715=1}
> 16:52:01,718 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.20.111.2
> :1620 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
> s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.20.111.2:1620, coord_addr=10
> .20.110.1:2715, is_server=true]]}
> 16:52:01,765 DEBUG ClientGmsImpl:113 - sending handleJoin(10.20.111.1:1172) to 1
> 0.40.10.52:4775
> 16:52:01,812 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.10.51:3426
> 16:52:01,812 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.20.111.2:1
> 620, coord_addr=10.20.110.1:2715, is_server=true]
> 16:52:01,812 DEBUG UDP:247 - received (mcast) 51 bytes from /10.40.124.1:64678 (
> size=51 bytes)
> 16:52:01,828 DEBUG UNICAST:222 - [10.20.111.1:1172] --> DATA(10.40.10.52:4775: #
> 82, first=false)
> 16:52:01,843 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.10.51
> :3426 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
> s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.10.51:3426, coord_addr=10
> .20.110.1:2715, is_server=true]]}
> 16:52:01,890 DEBUG UDP:757 - message is [dst: 231.12.21.132:45566, src: 10.40.12
> 4.1:64677 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCac
> heBus], PING=[PING: type=GET_MBRS_REQ, arg=null]}
> 16:52:01,906 DEBUG UDP:803 - sending msg to 10.40.10.52:4775 (src=10.20.111.1:11
> 72), headers are {UNICAST=[UNICAST: DATA, seqno=82], GMS=GmsHeader[JOIN_REQ]: mb
> r=10.20.111.1:1172, UDP=[UDP:channel_name=OSCacheBus]}
> 16:52:01,953 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.10.52:4775
> 16:52:01,953 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.10.51:3
> 426, coord_addr=10.20.110.1:2715, is_server=true]
> 16:52:01,984 DEBUG UDP:247 - received (mcast) 51 bytes from /10.20.111.1:1118 (s
> ize=51 bytes)
> 16:52:02,031 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.10.52
> :4775 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
> s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.10.52:4775, coord_addr=10
> .20.110.1:2715, is_server=true]]}
> 16:52:02,062 DEBUG PING:195 - received GET_MBRS_REQ from 10.40.124.1:64677, send
> ing response [PING: type=GET_MBRS_RSP, arg=[own_addr=10.20.111.1:1172, coord_add
> r=10.20.111.1:1172, is_server=false]]
> 16:52:02,078 DEBUG UDP:757 - message is [dst: 231.12.21.132:45566, src: 10.20.11
> 1.1:1117 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCach
> eBus], PING=[PING: type=GET_MBRS_REQ, arg=null]}
> 16:52:02,125 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.121.3:4249
> 16:52:02,156 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.10.52:4
> 775, coord_addr=10.20.110.1:2715, is_server=true]
> 16:52:02,156 DEBUG UDP:803 - sending msg to 10.40.124.1:64677 (src=10.20.111.1:1
> 172), headers are {PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.20.111.1:1172
> , coord_addr=10.20.111.1:1172, is_server=false]], UDP=[UDP:channel_name=OSCacheB
> us]}
> 16:52:02,187 DEBUG UDP:247 - received (mcast) 51 bytes from /10.40.124.1:64721 (
> size=51 bytes)
> 16:52:02,203 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.121.3
> :4249 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
> s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.121.3:4249, coord_addr=10
> .20.110.1:2715, is_server=true]]}
> 16:52:02,234 DEBUG PING:195 - received GET_MBRS_REQ from 10.20.111.1:1117, sendi
> ng response [PING: type=GET_MBRS_RSP, arg=[own_addr=10.20.111.1:1172, coord_addr
> =10.20.111.1:1172, is_server=false]]
> 16:52:02,296 DEBUG UDP:757 - message is [dst: 231.12.21.132:45566, src: 10.40.12
> 4.1:64720 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCac
> heBus], PING=[PING: type=GET_MBRS_REQ, arg=null]}
> 16:52:02,343 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.125.3:1739
> 16:52:02,375 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.121.3:4
> 249, coord_addr=10.20.110.1:2715, is_server=true]
> 16:52:02,375 DEBUG UDP:803 - sending msg to 10.20.111.1:1117 (src=10.20.111.1:11
> 72), headers are {PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.20.111.1:1172,
> coord_addr=10.20.111.1:1172, is_server=false]], UDP=[UDP:channel_name=OSCacheBu
> s]}
> 16:52:02,421 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.125.3
> :1739 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
> s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.125.3:1739, coord_addr=10
> .20.110.1:2715, is_server=true]]}
> 16:52:02,437 DEBUG PING:195 - received GET_MBRS_REQ from 10.40.124.1:64720, send
> ing response [PING: type=GET_MBRS_RSP, arg=[own_addr=10.20.111.1:1172, coord_add
> r=10.20.111.1:1172, is_server=false]]
> 16:52:02,531 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.126.2:4228
> 16:52:02,562 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.125.3:1
> 739, coord_addr=10.20.110.1:2715, is_server=true]
> 16:52:02,562 DEBUG UDP:803 - sending msg to 10.40.124.1:64720 (src=10.20.111.1:1
> 172), headers are {PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.20.111.1:1172
> , coord_addr=10.20.111.1:1172, is_server=false]], UDP=[UDP:channel_name=OSCacheB
> us]}
> 16:52:02,578 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.126.2
> :4228 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
> s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.126.2:4228, coord_addr=10
> .20.110.1:2715, is_server=true]]}
> 16:52:02,687 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.126.2:4
> 228, coord_addr=10.20.110.1:2715, is_server=true]
> 16:52:02,687 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.124.1:64677
> 16:52:02,718 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.124.1
> :64677 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheB
> us], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.124.1:64677, coord_addr=
> 10.40.124.1:64677, is_server=false]]}
> 16:52:02,765 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.124.2:31634
> 16:52:02,765 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.124.1:6
> 4677, coord_addr=10.40.124.1:64677, is_server=false]
> 16:52:02,781 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.124.2
> :31634 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheB
> us], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.124.2:31634, coord_addr=
> 10.20.110.1:2715, is_server=true]]}
> 16:52:02,843 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.124.2:3
> 1634, coord_addr=10.20.110.1:2715, is_server=true]
> 16:52:02,843 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.121.2:1403
> 16:52:02,875 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.121.2
> :1403 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
> s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.121.2:1403, coord_addr=10
> .20.110.1:2715, is_server=true]]}
> 16:52:02,906 DEBUG UDP:247 - received (mcast) 420 bytes from /10.40.121.3:4250 (
> size=420 bytes)
> 16:52:02,921 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.120.3:4838
> 16:52:02,921 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.121.2:1
> 403, coord_addr=10.20.110.1:2715, is_server=true]
> 16:52:02,937 DEBUG UDP:757 - message is [dst: 231.12.21.132:45566, src: 10.40.12
> 1.3:4249 (2 headers), size = 355 bytes], headers are {NAKACK=[NAKACK: MSG, seqno
> =601, range=null], UDP=[UDP:channel_name=OSCacheBus]}
> 16:52:02,953 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.120.3
> :4838 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
> s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.120.3:4838, coord_addr=10
> .20.110.1:2715, is_server=true]]}
> 16:52:03,015 DEBUG UDP:247 - received (mcast) 420 bytes from /10.40.121.3:4250 (
> size=420 bytes)
> 16:52:03,015 DEBUG NAKACK:340 - message was discarded (not yet server)
> 16:52:03,046 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.126.3:4204
> 16:52:03,046 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.120.3:4
> 838, coord_addr=10.20.110.1:2715, is_server=true]
> 16:52:03,078 DEBUG UDP:757 - message is [dst: 231.12.21.132:45566, src: 10.40.12
> 1.3:4249 (2 headers), size = 355 bytes], headers are {NAKACK=[NAKACK: MSG, seqno
> =602, range=null], UDP=[UDP:channel_name=OSCacheBus]}
> 16:52:03,093 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.126.3
> :4204 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
> s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.126.3:4204, coord_addr=10
> .20.110.1:2715, is_server=true]]}
> 16:52:03,156 DEBUG NAKACK:340 - message was discarded (not yet server)
> 16:52:03,203 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.126.1:58260
> 16:52:03,203 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.126.3:4
> 204, coord_addr=10.20.110.1:2715, is_server=true]
> 16:52:03,218 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.126.1
> :58260 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheB
> us], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.126.1:58260, coord_addr=
> 10.20.110.1:2715, is_server=true]]}
> 16:52:03,296 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.126.1:5
> 8260, coord_addr=10.20.110.1:2715, is_server=true]
> 16:52:03,296 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.125.2:3377
> 16:52:03,328 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.125.2
> :3377 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
> s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.125.2:3377, coord_addr=10
> .20.110.1:2715, is_server=true]]}
> 16:52:03,375 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.124.4:1764
> 16:52:03,375 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.125.2:3
> 377, coord_addr=10.20.110.1:2715, is_server=true]
> 16:52:03,390 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.124.4
> :1764 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
> s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.124.4:1764, coord_addr=10
> .20.110.1:2715, is_server=true]]}
> 16:52:03,453 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.121.4:1832
> 16:52:03,453 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.124.4:1
> 764, coord_addr=10.20.110.1:2715, is_server=true]
> 16:52:03,468 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.121.4
> :1832 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
> s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.121.4:1832, coord_addr=10
> .20.110.1:2715, is_server=true]]}
> 16:52:03,531 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.121.4:1
> 832, coord_addr=10.20.110.1:2715, is_server=true]
> 16:52:03,531 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.125.4:2799
> 16:52:03,562 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.125.4
> :2799 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
> s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.125.4:2799, coord_addr=10
> .20.110.1:2715, is_server=true]]}
> 16:52:03,609 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.125.4:2
> 799, coord_addr=10.20.110.1:2715, is_server=true]
> 16:52:03,609 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.120.4:3696
> 16:52:03,640 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.120.4
> :3696 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
> s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.120.4:3696, coord_addr=10
> .20.110.1:2715, is_server=true]]}
> 16:52:03,687 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.120.4:3
> 696, coord_addr=10.20.110.1:2715, is_server=true]
> 16:52:03,687 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.120.2:2013
> 16:52:03,718 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.120.2
> :2013 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
> s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.120.2:2013, coord_addr=10
> .20.110.1:2715, is_server=true]]}
> 16:52:03,765 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.126.4:21141
> 16:52:03,765 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.120.2:2
> 013, coord_addr=10.20.110.1:2715, is_server=true]
> 16:52:03,781 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.126.4
> :21141 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheB
> us], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.126.4:21141, coord_addr=
> 10.20.110.1:2715, is_server=true]]}
> 16:52:03,843 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.126.4:2
> 1141, coord_addr=10.20.110.1:2715, is_server=true]
> 16:52:03,843 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.124.3:15988
> 16:52:03,875 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.124.3
> :15988 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheB
> us], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.124.3:15988, coord_addr=
> 10.20.110.1:2715, is_server=true]]}
> 16:52:03,921 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.125.1:3612
> 16:52:03,921 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.124.3:1
> 5988, coord_addr=10.20.110.1:2715, is_server=true]
> 16:52:03,937 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.125.1
> :3612 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
> s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.125.1:3612, coord_addr=10
> .20.110.1:2715, is_server=true]]}
> 16:52:04,000 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.125.1:3
> 612, coord_addr=10.20.110.1:2715, is_server=true]
> 16:52:04,000 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.124.1:64720
> 16:52:04,031 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.124.1
> :64720 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheB
> us], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.124.1:64720, coord_addr=
> 10.40.124.1:64720, is_server=false]]}
> 16:52:04,078 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.121.1:3810
> 16:52:04,078 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.124.1:6
> 4720, coord_addr=10.40.124.1:64720, is_server=false]
> 16:52:04,093 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.121.1
> :3810 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
> s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.121.1:3810, coord_addr=10
> .20.110.1:2715, is_server=true]]}
> 16:52:04,156 DEBUG UDP:1542 - received (ucast) 62 bytes from /10.40.10.52:4775
> 16:52:04,156 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.121.1:3
> 810, coord_addr=10.20.110.1:2715, is_server=true]
> 16:52:04,171 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.10.52
> :4775 (2 headers), size = 0 bytes], headers are {UNICAST=[UNICAST: DATA_ACK, seq
> no=82], UDP=[UDP:channel_name=OSCacheBus]}
> 16:52:04,218 DEBUG UDP:1542 - received (ucast) 82 bytes from /10.40.10.52:4775
> 16:52:04,218 DEBUG UNICAST:387 - [10.20.111.1:1172] <-- ACK(10.40.10.52:4775: #8
> 2)
> 16:52:04,234 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.10.52
> :4775 (3 headers), size = 0 bytes], headers are {UNICAST=[UNICAST: DATA, seqno=6
> 3], GMS=GmsHeader[JOIN_RSP]: join_rsp=null, UDP=[UDP:channel_name=OSCacheBus]}
> 16:52:04,281 DEBUG UNICAST:402 - [10.20.111.1:1172] --> ACK(10.40.10.52:4775: #6
> 3)
> 16:52:04,312 DEBUG UNICAST:344 - [10.20.111.1:1172] <-- DATA(10.40.10.52:4775: #
> 63, first=false
> 16:52:04,312 DEBUG UDP:803 - sending msg to 10.40.10.52:4775 (src=10.20.111.1:11
> 72), headers are {UNICAST=[UNICAST: DATA_ACK, seqno=63], UDP=[UDP:channel_name=O
> SCacheBus]}
> 16:52:04,328 WARN ClientGmsImpl:118 - join(10.20.111.1:1172) failed, retrying
> 16:52:05,312 DEBUG UDP:247 - received (mcast) 420 bytes from /10.40.121.3:4250 (
> size=420 bytes)
> 16:52:05,343 DEBUG UDP:757 - message is [dst: 231.12.21.132:45566, src: 10.40.12
> 1.3:4249 (2 headers), size = 355 bytes], headers are {NAKACK=[NAKACK: MSG, seqno
> =603, range=null], UDP=[UDP:channel_name=OSCacheBus]}
> 16:52:05,375 DEBUG UDP:247 - received (mcast) 420 bytes from /10.40.121.3:4250 (
> size=420 bytes)
> 16:52:05,375 DEBUG NAKACK:340 - message was discarded (not yet server)
> 16:52:05,390 DEBUG UDP:757 - message is [dst: 231.12.21.132:45566, src: 10.40.12
> 1.3:4249 (2 headers), size = 355 bytes], headers are {NAKACK=[NAKACK: MSG, seqno
> =604, range=null], UDP=[UDP:channel_name=OSCacheBus]}
> 16:52:05,437 DEBUG NAKACK:340 - message was discarded (not yet server)
> 16:52:06,375 DEBUG PingWaiter:95 - clearing old responses: [[own_addr=10.20.111.
> 1:1117, coord_addr=10.20.111.1:1117, is_server=false], [own_addr=10.20.111.3:651
> 80, coord_addr=10.40.10.52:4775, is_server=true], [own_addr=10.20.110.2:56053, c
> oord_addr=10.20.110.1:2715, is_server=true], [own_addr=10.20.110.3:45282, coord_
> addr=10.40.10.52:4775, is_server=true], [own_addr=10.20.111.2:1620, coord_addr=1
> 0.20.110.1:2715, is_server=true], [own_addr=10.40.10.51:3426, coord_addr=10.20.1
> 10.1:2715, is_server=true], [own_addr=10.40.10.52:4775, coord_addr=10.20.110.1:2
> 715, is_server=true], [own_addr=10.40.121.3:4249, coord_addr=10.20.110.1:2715, i
> s_server=true], [own_addr=10.40.125.3:1739, coord_addr=10.20.110.1:2715, is_serv
> er=true], [own_addr=10.40.126.2:4228, coord_addr=10.20.110.1:2715, is_server=tru
> e], [own_addr=10.40.124.1:64677, coord_addr=10.40.124.1:64677, is_server=false],
> [own_addr=10.40.124.2:31634, coord_addr=10.20.110.1:2715, is_server=true], [own
> _addr=10.40.121.2:1403, coord_addr=10.20.110.1:2715, is_server=true], [own_addr=
> 10.40.120.3:4838, coord_addr=10.20.110.1:2715, is_server=true], [own_addr=10.40.
> 126.3:4204, coord_addr=10.20.110.1:2715, is_server=true], [own_addr=10.40.126.1:
> 58260, coord_addr=10.20.110.1:2715, is_server=true], [own_addr=10.40.125.2:3377,
> coord_addr=10.20.110.1:2715, is_server=true], [own_addr=10.40.124.4:1764, coord
> _addr=10.20.110.1:2715, is_server=true], [own_addr=10.40.121.4:1832, coord_addr=
> 10.20.110.1:2715, is_server=true], [own_addr=10.40.125.4:2799, coord_addr=10.20.
> 110.1:2715, is_server=true], [own_addr=10.40.120.4:3696, coord_addr=10.20.110.1:
> 2715, is_server=true], [own_addr=10.40.120.2:2013, coord_addr=10.20.110.1:2715,
> is_server=true], [own_addr=10.40.126.4:21141, coord_addr=10.20.110.1:2715, is_se
> rver=true], [own_addr=10.40.124.3:15988, coord_addr=10.20.110.1:2715, is_server=
> true], [own_addr=10.40.125.1:3612, coord_addr=10.20.110.1:2715, is_server=true],
> [own_addr=10.40.124.1:64720, coord_addr=10.40.124.1:64720, is_server=false], [o
> wn_addr=10.40.121.1:3810, coord_addr=10.20.110.1:2715, is_server=true]]
> 16:52:06,640 DEBUG PingWaiter:107 - waiting for initial members: time_to_wait=20
> 00, got 0 rsps
> 16:52:06,640 DEBUG PingSender:57 - sending GET_MBRS_REQ
> 16:52:06,671 DEBUG UDP:803 - sending msg to 231.12.21.132:45566 (src=10.20.111.1
> :1172), headers are {PING=[PING: type=GET_MBRS_REQ, arg=null], UDP=[UDP:channel_
> name=OSCacheBus]}
> 16:52:06,703 DEBUG UDP:820 - looped back local message [dst: 231.12.21.132:45566
> , src: 10.20.111.1:1172 (2 headers), size = 0 bytes]
> 16:52:06,734 DEBUG PING:183 - discarded my own discovery request
> 16:52:06,734 DEBUG UDP:247 - received (mcast) 51 bytes from /10.20.111.1:1173 (s
> ize=51 bytes)
> 16:52:06,734 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.20.111.1:1117
> 16:52:06,765 DEBUG UDP:748 - discarded own loopback multicast packet
> 16:52:06,781 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.20.111.1
> :1117 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
> s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.20.111.1:1117, coord_addr=10
> .20.111.1:1117, is_server=false]]}
> 16:52:06,828 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.20.111.3:65180
> 16:52:06,828 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.20.111.1:1
> 117, coord_addr=10.20.111.1:1117, is_server=false]
> 16:52:06,843 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.20.111.3
> :65180 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheB
> us], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.20.111.3:65180, coord_addr=
> 10.40.10.52:4775, is_server=true]]}
> 16:52:06,859 DEBUG PingWaiter:107 - waiting for initial members: time_to_wait=17
> 81, got 1 rsps
> 16:52:06,906 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.20.110.3:45282
> 16:52:06,906 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.20.111.3:6
> 5180, coord_addr=10.40.10.52:4775, is_server=true]
> 16:52:06,937 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.20.110.3
> :45282 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheB
> us], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.20.110.3:45282, coord_addr=
> 10.40.10.52:4775, is_server=true]]}
> 16:52:06,968 DEBUG PingWaiter:107 - waiting for initial members: time_to_wait=16
> 72, got 2 rsps
> 16:52:07,000 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.20.111.2:1620
> 16:52:07,000 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.20.110.3:4
> 5282, coord_addr=10.40.10.52:4775, is_server=true]
> 16:52:07,031 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.20.111.2
> :1620 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
> s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.20.111.2:1620, coord_addr=10
> .20.110.1:2715, is_server=true]]}
> 16:52:07,062 DEBUG PingWaiter:122 - initial mbrs are [[own_addr=10.20.111.1:1117
> , coord_addr=10.20.111.1:1117, is_server=false], [own_addr=10.20.111.3:65180, co
> ord_addr=10.40.10.52:4775, is_server=true], [own_addr=10.20.110.3:45282, coord_a
> ddr=10.40.10.52:4775, is_server=true]]
> 16:52:07,109 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.10.52:4775
> 16:52:07,109 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.20.111.2:1
> 620, coord_addr=10.20.110.1:2715, is_server=true]
> 16:52:07,156 DEBUG ClientGmsImpl:66 - initial_mbrs are [[own_addr=10.20.111.1:11
> 17, coord_addr=10.20.111.1:1117, is_server=false], [own_addr=10.20.111.3:65180,
> coord_addr=10.40.10.52:4775, is_server=true], [own_addr=10.20.110.3:45282, coord
> _addr=10.40.10.52:4775, is_server=true]]
> 16:52:07,156 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.10.52
> :4775 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
> s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.10.52:4775, coord_addr=10
> .20.110.1:2715, is_server=true]]}
> 16:52:07,234 DEBUG ClientGmsImpl:113 - sending handleJoin(10.20.111.1:1172) to 1
> 0.40.10.52:4775
> 16:52:07,265 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.125.3:1739
> 16:52:07,265 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.10.52:4
> 775, coord_addr=10.20.110.1:2715, is_server=true]
> 16:52:07,296 DEBUG UNICAST:222 - [10.20.111.1:1172] --> DATA(10.40.10.52:4775: #
> 83, first=false)
> 16:52:07,312 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.125.3
> :1739 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
> s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.125.3:1739, coord_addr=10
> .20.110.1:2715, is_server=true]]}
> 16:52:07,359 DEBUG UDP:803 - sending msg to 10.40.10.52:4775 (src=10.20.111.1:11
> 72), headers are {UNICAST=[UNICAST: DATA, seqno=83], GMS=GmsHeader[JOIN_REQ]: mb
> r=10.20.111.1:1172, UDP=[UDP:channel_name=OSCacheBus]}
> 16:52:07,406 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.121.3:4249
> 16:52:07,406 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.125.3:1
> 739, coord_addr=10.20.110.1:2715, is_server=true]
> 16:52:07,453 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.121.3
> :4249 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
> s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.121.3:4249, coord_addr=10
> .20.110.1:2715, is_server=true]]}
> 16:52:07,515 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.10.51:3426
> 16:52:07,515 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.121.3:4
> 249, coord_addr=10.20.110.1:2715, is_server=true]
> 16:52:07,531 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.10.51
> :3426 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
> s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.10.51:3426, coord_addr=10
> .20.110.1:2715, is_server=true]]}
> 16:52:07,593 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.10.51:3
> 426, coord_addr=10.20.110.1:2715, is_server=true]
> 16:52:07,593 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.126.2:4228
> 16:52:07,625 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.126.2
> :4228 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
> s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.126.2:4228, coord_addr=10
> .20.110.1:2715, is_server=true]]}
> 16:52:07,671 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.126.1:58260
> 16:52:07,671 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.126.2:4
> 228, coord_addr=10.20.110.1:2715, is_server=true]
> 16:52:07,687 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.126.1
> :58260 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheB
> us], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.126.1:58260, coord_addr=
> 10.20.110.1:2715, is_server=true]]}
> 16:52:07,750 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.125.4:2799
> 16:52:07,750 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.126.1:5
> 8260, coord_addr=10.20.110.1:2715, is_server=true]
> 16:52:07,765 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.125.4
> :2799 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
> s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.125.4:2799, coord_addr=10
> .20.110.1:2715, is_server=true]]}
> 16:52:07,828 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.125.4:2
> 799, coord_addr=10.20.110.1:2715, is_server=true]
> 16:52:07,828 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.121.1:3810
> 16:52:07,859 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.121.1
> :3810 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
> s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.121.1:3810, coord_addr=10
> .20.110.1:2715, is_server=true]]}
> Terminate batch job (Y/N)? y
> D:\sbroot\r6.1.0.0\Bin>
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
More information about the jboss-jira
mailing list