New subject: [JBoss JIRA] Commented: (JGRP-343) Unable to bring additional node to a cluster due to WARN ClientGmsImpl:331 - there was more than 1 candidate for coord and WARN ClientGmsImpl:118 - join(10.20.111.1:1362) failed, retrying

Monday, 16 October 2006

Unable to bring additional node to a cluster due to WARN ClientGmsImpl:331 - there was
more than 1 candidate for coord and WARN ClientGmsImpl:118 - join(10.20.111.1:1362)
failed, retrying
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

                 Key: JGRP-343
                 URL: http://jira.jboss.com/jira/browse/JGRP-343
             Project: JGroups
          Issue Type: Bug
    Affects Versions: 2.2.8
         Environment: Win2k, Linux RedHat 
Linux qacluster01 2.4.21-32.0.1.ELsmp #1 SMP Tue May 17 17:52:23 EDT 2005 i686 i686 i386
GNU/Linux

            Reporter: Maciek Kolesnik
         Assigned To: Bela Ban

We have an environment that consists of approx 20-30 nodes all running Hibernate
3.1.1->OSCache->Jgroups 2.2.8

In some cases the state of the cluster becomes inconsitent due to multiple nodes being
rebooted at random times and this results in some nodes thnking that Coordinator is NODE-X
and other nodes thinking that Coordinator is NODE-Y.  It is actually very easy to repsoduc
it in this 20-node environment by rebooting the coordinator node.

The confused nodes never recover from the wrong state even when NODE-X goes off-line. The
only way to correct is to reboot all ~20 nodes which really sucks

During the confused cluster state i am unable to bring additional nodes up - the JOIN
error keeps scrolling and they never become part of the cluster:

19:20:04,500  WARN ClientGmsImpl:118 - join(10.20.111.1:1362) failed, retrying
...

With debug on we were able to determine that the reason for it is the following:

16:51:56,203  WARN ClientGmsImpl:331 - there was more than 1 candidate for coord
inator: {10.40.10.52:4775=1, 10.20.110.1:2715=1}    

This was when node 10.20.110.1 has been dead for at least an hour.

The state of the cluster (each nodes' oppinion on who should be the coordinator is
nicely reported here:

16:52:06,375 DEBUG PingWaiter:95 - clearing old responses: [[own_addr=10.20.111.
1:1117, coord_addr=10.20.111.1:1117, is_server=false], [own_addr=10.20.111.3:651
80, coord_addr=10.40.10.52:4775, is_server=true], [own_addr=10.20.110.2:56053, c
oord_addr=10.20.110.1:2715, is_server=true], [own_addr=10.20.110.3:45282, coord_
addr=10.40.10.52:4775, is_server=true], [own_addr=10.20.111.2:1620, coord_addr=1
0.20.110.1:2715, is_server=true], [own_addr=10.40.10.51:3426, coord_addr=10.20.1
10.1:2715, is_server=true], [own_addr=10.40.10.52:4775, coord_addr=10.20.110.1:2
715, is_server=true], [own_addr=10.40.121.3:4249, coord_addr=10.20.110.1:2715, i
s_server=true], [own_addr=10.40.125.3:1739, coord_addr=10.20.110.1:2715, is_serv
er=true], [own_addr=10.40.126.2:4228, coord_addr=10.20.110.1:2715, is_server=tru
e], [own_addr=10.40.124.1:64677, coord_addr=10.40.124.1:64677, is_server=false],
 [own_addr=10.40.124.2:31634, coord_addr=10.20.110.1:2715, is_server=true], [own
_addr=10.40.121.2:1403, coord_addr=10.20.110.1:2715, is_server=true], [own_addr=
10.40.120.3:4838, coord_addr=10.20.110.1:2715, is_server=true], [own_addr=10.40.
126.3:4204, coord_addr=10.20.110.1:2715, is_server=true], [own_addr=10.40.126.1:
58260, coord_addr=10.20.110.1:2715, is_server=true], [own_addr=10.40.125.2:3377,
 coord_addr=10.20.110.1:2715, is_server=true], [own_addr=10.40.124.4:1764, coord
_addr=10.20.110.1:2715, is_server=true], [own_addr=10.40.121.4:1832, coord_addr=
10.20.110.1:2715, is_server=true], [own_addr=10.40.125.4:2799, coord_addr=10.20.
110.1:2715, is_server=true], [own_addr=10.40.120.4:3696, coord_addr=10.20.110.1:
2715, is_server=true], [own_addr=10.40.120.2:2013, coord_addr=10.20.110.1:2715,
is_server=true], [own_addr=10.40.126.4:21141, coord_addr=10.20.110.1:2715, is_se
rver=true], [own_addr=10.40.124.3:15988, coord_addr=10.20.110.1:2715, is_server=
true], [own_addr=10.40.125.1:3612, coord_addr=10.20.110.1:2715, is_server=true],
 [own_addr=10.40.124.1:64720, coord_addr=10.40.124.1:64720, is_server=false], [o
wn_addr=10.40.121.1:3810, coord_addr=10.20.110.1:2715, is_server=true]]

Majority of the nodes think that our DEAD node should be the Coordinator.

And finally here is the full debug trace from an attempt to bring a new node into the
cluster:

D:\sbroot\r6.1.0.0\Bin>startEmulator.bat
16:51:49,578 DEBUG ClassConfigurator:97 - mapping is:
1:      class org.jgroups.stack.IpAddress
2:      class org.jgroups.protocols.CAUSAL$CausalHeader
3:      class org.jgroups.protocols.FD$FdHeader
4:      class org.jgroups.protocols.FD_PID$FdHeader
5:      class org.jgroups.protocols.FD_PROB$FdHeader
6:      class org.jgroups.protocols.FD_SOCK$FdHeader
7:      class org.jgroups.protocols.FragHeader
8:      class org.jgroups.protocols.MERGE$MergeHeader
9:      class org.jgroups.protocols.NakAckHeader
10:     class org.jgroups.protocols.PARTITIONER$PartitionerHeader
11:     class org.jgroups.protocols.PerfHeader
12:     class org.jgroups.protocols.PIGGYBACK$PiggybackHeader
13:     class org.jgroups.protocols.PingHeader
14:     class org.jgroups.protocols.TcpHeader
15:     class org.jgroups.protocols.TOTAL$Header
16:     class org.jgroups.protocols.TOTAL_OLD$TotalHeader
17:     class org.jgroups.protocols.TOTAL_TOKEN$TotalTokenHeader
18:     class org.jgroups.protocols.TOTAL_TOKEN$RingTokenHeader
19:     class org.jgroups.protocols.TunnelHeader
20:     class org.jgroups.protocols.UdpHeader
21:     class org.jgroups.protocols.UNICAST$UnicastHeader
22:     class org.jgroups.protocols.VERIFY_SUSPECT$VerifyHeader
23:     class org.jgroups.protocols.WANPIPE$WanPipeHeader
24:     class org.jgroups.protocols.pbcast.GMS$GmsHeader
25:     class org.jgroups.protocols.pbcast.NakAckHeader
26:     class org.jgroups.protocols.pbcast.PbcastHeader
27:     class org.jgroups.protocols.pbcast.STABLE$StableHeader
28:     class org.jgroups.protocols.pbcast.STATE_TRANSFER$StateHeader
29:     class org.jgroups.protocols.SMACK$SmackHeader
30:     class org.jgroups.Message
31:     class org.jgroups.View
32:     class org.jgroups.ViewId
33:     class org.jgroups.util.List
34:     interface org.jgroups.Address
35:     class [Lorg.jgroups.Address;
36:     class org.jgroups.protocols.PingRsp
37:     class [Ljava.lang.Object;
38:     class java.util.Vector
39:     class org.jgroups.protocols.pbcast.JoinRsp
40:     class org.jgroups.protocols.pbcast.Digest
41:     class java.util.Hashtable
50:     class org.jgroups.service.lease.LeaseRequestHeader
51:     class org.jgroups.service.lease.LeaseResponseHeader
52:     class org.jgroups.service.lease.DenyResponseHeader
53:     class org.jgroups.protocols.COMPRESS$CompressHeader
54:     class org.jgroups.protocols.FC$FcHeader
55:     class org.jgroups.protocols.WanPipeAddress

16:51:49,765 DEBUG AUTOCONF:166 - frag_size=64000
16:51:49,843 DEBUG GMS:93 - changed role to org.jgroups.protocols.pbcast.ClientG
msImpl
16:51:49,890 DEBUG UDP:359 - creating sockets and starting threads
16:51:49,906  INFO UDP:1048 - sockets will use interface 10.20.111.1
16:51:49,921  INFO UDP:1094 - socket information:
local_addr=10.20.111.1:1172, mcast_addr=231.12.21.132:45566, bind_addr=/10.20.11
1.1, ttl=32
sock: bound to 10.20.111.1:1172, receive buffer size=64000, send buffer size=320
00
mcast_recv_sock: bound to 10.20.111.1:45566, send buffer size=150000, receive bu
ffer size=80000
mcast_send_sock: bound to 10.20.111.1:1173, send buffer size=150000, receive buf
fer size=80000

-------------------------------------------------------
GMS: address is 10.20.111.1:1172
-------------------------------------------------------
16:51:50,015 DEBUG UDP:1347 - created unicast receiver thread
16:51:50,062 DEBUG FD_SOCK:872 - waiting for client connections on 0.0.0.0/0.0.0
.0:1174
16:51:50,062 DEBUG PingWaiter:107 - waiting for initial members: time_to_wait=20
00, got 0 rsps
16:51:50,062 DEBUG PingSender:57 - sending GET_MBRS_REQ
16:51:50,125 DEBUG UDP:803 - sending msg to 231.12.21.132:45566 (src=10.20.111.1
:1172), headers are {PING=[PING: type=GET_MBRS_REQ, arg=null], UDP=[UDP:channel_
name=OSCacheBus]}
16:51:50,156 DEBUG UDP:820 - looped back local message [dst: 231.12.21.132:45566
, src: 10.20.111.1:1172 (2 headers), size = 0 bytes]
16:51:50,187 DEBUG PING:183 - discarded my own discovery request
16:51:50,187 DEBUG UDP:247 - received (mcast) 51 bytes from /10.20.111.1:1173 (s
ize=51 bytes)
16:51:50,187 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.20.111.1:1117
16:51:50,218 DEBUG UDP:748 - discarded own loopback multicast packet
16:51:50,234 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.20.111.1
:1117 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.20.111.1:1117, coord_addr=10
.20.111.1:1117, is_server=false]]}
16:51:50,281 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.20.111.3:65180
16:51:50,281 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.20.111.1:1
117, coord_addr=10.20.111.1:1117, is_server=false]
16:51:50,296 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.20.111.3
:65180 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheB
us], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.20.111.3:65180, coord_addr=
10.40.10.52:4775, is_server=true]]}
16:51:50,312 DEBUG PingWaiter:107 - waiting for initial members: time_to_wait=17
50, got 1 rsps
16:51:50,359 DEBUG UDP:247 - received (mcast) 420 bytes from /10.40.124.4:1765 (
size=420 bytes)
16:51:50,359 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.20.111.3:6
5180, coord_addr=10.40.10.52:4775, is_server=true]
16:51:50,359 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.20.111.2:1620
16:51:50,406 DEBUG UDP:757 - message is [dst: 231.12.21.132:45566, src: 10.40.12
4.4:1764 (2 headers), size = 355 bytes], headers are {NAKACK=[NAKACK: MSG, seqno
=537, range=null], UDP=[UDP:channel_name=OSCacheBus]}
16:51:50,421 DEBUG PingWaiter:107 - waiting for initial members: time_to_wait=16
41, got 2 rsps
16:51:50,437 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.20.111.2
:1620 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.20.111.2:1620, coord_addr=10
.20.110.1:2715, is_server=true]]}
16:51:50,484 DEBUG UDP:247 - received (mcast) 420 bytes from /10.40.124.4:1765 (
size=420 bytes)
16:51:50,484 DEBUG NAKACK:340 - message was discarded (not yet server)
16:51:50,546 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.20.110.3:45282
16:51:50,546 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.20.111.2:1
620, coord_addr=10.20.110.1:2715, is_server=true]
16:51:50,562 DEBUG UDP:757 - message is [dst: 231.12.21.132:45566, src: 10.40.12
4.4:1764 (2 headers), size = 355 bytes], headers are {NAKACK=[NAKACK: MSG, seqno
=538, range=null], UDP=[UDP:channel_name=OSCacheBus]}
16:51:50,593 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.20.110.3
:45282 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheB
us], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.20.110.3:45282, coord_addr=
10.40.10.52:4775, is_server=true]]}
16:51:50,609 DEBUG PingWaiter:122 - initial mbrs are [[own_addr=10.20.111.1:1117
, coord_addr=10.20.111.1:1117, is_server=false], [own_addr=10.20.111.3:65180, co
ord_addr=10.40.10.52:4775, is_server=true], [own_addr=10.20.111.2:1620, coord_ad
dr=10.20.110.1:2715, is_server=true]]
16:51:50,640 DEBUG NAKACK:340 - message was discarded (not yet server)
16:51:50,687 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.20.110.2:56053
16:51:50,687 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.20.110.3:4
5282, coord_addr=10.40.10.52:4775, is_server=true]
16:51:50,750 DEBUG ClientGmsImpl:66 - initial_mbrs are [[own_addr=10.20.111.1:11
17, coord_addr=10.20.111.1:1117, is_server=false], [own_addr=10.20.111.3:65180,
coord_addr=10.40.10.52:4775, is_server=true], [own_addr=10.20.111.2:1620, coord_
addr=10.20.110.1:2715, is_server=true]]
16:51:50,765 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.20.110.2
:56053 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheB
us], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.20.110.2:56053, coord_addr=
10.20.110.1:2715, is_server=true]]}
16:51:50,828  WARN ClientGmsImpl:331 - there was more than 1 candidate for coord
inator: {10.40.10.52:4775=1, 10.20.110.1:2715=1}
16:51:50,875 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.10.52:4775
16:51:50,875 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.20.110.2:5
6053, coord_addr=10.20.110.1:2715, is_server=true]
16:51:50,890 DEBUG ClientGmsImpl:113 - sending handleJoin(10.20.111.1:1172) to 1
0.40.10.52:4775
16:51:50,906 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.10.52
:4775 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.10.52:4775, coord_addr=10
.20.110.1:2715, is_server=true]]}
16:51:50,953 DEBUG UNICAST:222 - [10.20.111.1:1172] --> DATA(10.40.10.52:4775: #
80, first=true)
16:51:51,000 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.10.52:4
775, coord_addr=10.20.110.1:2715, is_server=true]
16:51:51,000 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.10.51:3426
16:51:51,015 DEBUG STABLE:324 - stable task started; num_gossip_runs=3, max_goss
ip_runs=3
16:51:51,046 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.10.51
:3426 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.10.51:3426, coord_addr=10
.20.110.1:2715, is_server=true]]}
16:51:51,062 DEBUG UDP:803 - sending msg to 10.40.10.52:4775 (src=10.20.111.1:11
72), headers are {UNICAST=[UNICAST: DATA, seqno=80], GMS=GmsHeader[JOIN_REQ]: mb
r=10.20.111.1:1172, UDP=[UDP:channel_name=OSCacheBus]}
16:51:51,109 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.126.2:4228
16:51:51,109 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.10.51:3
426, coord_addr=10.20.110.1:2715, is_server=true]
16:51:51,156 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.126.2
:4228 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.126.2:4228, coord_addr=10
.20.110.1:2715, is_server=true]]}
16:51:51,234 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.120.3:4838
16:51:51,234 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.126.2:4
228, coord_addr=10.20.110.1:2715, is_server=true]
16:51:51,234 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.120.3
:4838 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.120.3:4838, coord_addr=10
.20.110.1:2715, is_server=true]]}
16:51:51,296 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.126.1:58260
16:51:51,296 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.120.3:4
838, coord_addr=10.20.110.1:2715, is_server=true]
16:51:51,312 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.126.1
:58260 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheB
us], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.126.1:58260, coord_addr=
10.20.110.1:2715, is_server=true]]}
16:51:51,390 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.120.4:3696
16:51:51,390 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.126.1:5
8260, coord_addr=10.20.110.1:2715, is_server=true]
16:51:51,390 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.120.4
:3696 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.120.4:3696, coord_addr=10
.20.110.1:2715, is_server=true]]}
16:51:51,453 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.121.1:3810
16:51:51,453 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.120.4:3
696, coord_addr=10.20.110.1:2715, is_server=true]
16:51:51,468 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.121.1
:3810 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.121.1:3810, coord_addr=10
.20.110.1:2715, is_server=true]]}
16:51:51,531 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.121.1:3
810, coord_addr=10.20.110.1:2715, is_server=true]
16:51:51,531 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.121.3:4249
16:51:51,578 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.121.3
:4249 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.121.3:4249, coord_addr=10
.20.110.1:2715, is_server=true]]}
16:51:51,609 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.124.1:64677
16:51:51,609 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.121.3:4
249, coord_addr=10.20.110.1:2715, is_server=true]
16:51:51,625 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.124.1
:64677 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheB
us], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.124.1:64677, coord_addr=
10.40.124.1:64677, is_server=false]]}
16:51:51,687 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.125.1:3612
16:51:51,687 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.124.1:6
4677, coord_addr=10.40.124.1:64677, is_server=false]
16:51:51,703 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.125.1
:3612 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.125.1:3612, coord_addr=10
.20.110.1:2715, is_server=true]]}
16:51:51,765 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.125.1:3
612, coord_addr=10.20.110.1:2715, is_server=true]
16:51:51,765 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.124.4:1764
16:51:51,796 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.124.4
:1764 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.124.4:1764, coord_addr=10
.20.110.1:2715, is_server=true]]}
16:51:51,843 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.126.3:4204
16:51:51,843 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.124.4:1
764, coord_addr=10.20.110.1:2715, is_server=true]
16:51:51,859 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.126.3
:4204 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.126.3:4204, coord_addr=10
.20.110.1:2715, is_server=true]]}
16:51:51,921 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.121.2:1403
16:51:51,921 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.126.3:4
204, coord_addr=10.20.110.1:2715, is_server=true]
16:51:51,937 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.121.2
:1403 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.121.2:1403, coord_addr=10
.20.110.1:2715, is_server=true]]}
16:51:52,000 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.121.2:1
403, coord_addr=10.20.110.1:2715, is_server=true]
16:51:52,000 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.121.4:1832
16:51:52,031 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.121.4
:1832 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.121.4:1832, coord_addr=10
.20.110.1:2715, is_server=true]]}
16:51:52,062 DEBUG UDP:247 - received (mcast) 1098 bytes from /10.40.125.2:3378
(size=1098 bytes)
16:51:52,078 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.120.2:2013
16:51:52,078 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.121.4:1
832, coord_addr=10.20.110.1:2715, is_server=true]
16:51:52,093 DEBUG UDP:757 - message is [dst: 231.12.21.132:45566, src: 10.40.12
5.2:3377 (3 headers), size = 0 bytes], headers are {STABLE=[STABLE_GOSSIP]: dige
st is [10.20.110.1:2715: [0 : 595 (595)], 10.20.110.2:56053: [1 : 1490 (1490)],
10.20.110.3:45282: [0 : 630 (630)], 10.20.111.1:53038: [0 : 21 (21)], 10.20.111.
2:1620: [0 : 952 (952)], 10.20.111.3:65180: [0 : 379 (379)], 10.40.10.51:3426: [
5 : 3906 (3906)], 10.40.10.52:4775: [16 : 3457 (3457)], 10.40.120.2:2013: [0 : 6
78 (678)], 10.40.120.3:4838: [0 : 674 (674)], 10.40.120.4:3696: [0 : 658 (658)],
 10.40.121.1:3810: [0 : 884 (884)], 10.40.121.2:1403: [0 : 631 (631)], 10.40.121
.3:4249: [0 : 598 (598)], 10.40.121.4:1832: [0 : 773 (773)], 10.40.124.1:55644:
[0 : 512 (512)], 10.40.124.2:31634: [0 : 652 (652)], 10.40.124.3:15988: [0 : 790
 (790)], 10.40.124.4:1764: [0 : 538 (538)], 10.40.125.1:3612: [0 : 804 (804)], 1
0.40.125.2:3377: [0 : 539 (539)], 10.40.125.3:1739: [0 : 774 (774)], 10.40.125.4
:2799: [0 : 756 (756)], 10.40.126.1:58260: [0 : 52 (52)], 10.40.126.2:4228: [0 :
 50 (50)], 10.40.126.3:4204: [0 : 51 (51)], 10.40.126.4:21141: [0 : 643 (643)]],
 NAKACK=[NAKACK: MSG, seqno=540, range=null], UDP=[UDP:channel_name=OSCacheBus]}

16:51:52,109 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.120.2
:2013 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.120.2:2013, coord_addr=10
.20.110.1:2715, is_server=true]]}
16:51:52,296 DEBUG NAKACK:340 - message was discarded (not yet server)
16:51:52,343 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.120.2:2
013, coord_addr=10.20.110.1:2715, is_server=true]
16:51:52,343 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.124.3:15988
16:51:52,390 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.124.3
:15988 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheB
us], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.124.3:15988, coord_addr=
10.20.110.1:2715, is_server=true]]}
16:51:52,437 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.124.3:1
5988, coord_addr=10.20.110.1:2715, is_server=true]
16:51:52,437 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.125.4:2799
16:51:52,468 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.125.4
:2799 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.125.4:2799, coord_addr=10
.20.110.1:2715, is_server=true]]}
16:51:52,515 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.125.2:3377
16:51:52,515 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.125.4:2
799, coord_addr=10.20.110.1:2715, is_server=true]
16:51:52,531 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.125.2
:3377 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.125.2:3377, coord_addr=10
.20.110.1:2715, is_server=true]]}
16:51:52,593 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.124.2:31634
16:51:52,593 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.125.2:3
377, coord_addr=10.20.110.1:2715, is_server=true]
16:51:52,609 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.124.2
:31634 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheB
us], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.124.2:31634, coord_addr=
10.20.110.1:2715, is_server=true]]}
16:51:52,671 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.126.4:21141
16:51:52,671 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.124.2:3
1634, coord_addr=10.20.110.1:2715, is_server=true]
16:51:52,687 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.126.4
:21141 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheB
us], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.126.4:21141, coord_addr=
10.20.110.1:2715, is_server=true]]}
16:51:52,750 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.124.1:64720
16:51:52,750 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.126.4:2
1141, coord_addr=10.20.110.1:2715, is_server=true]
16:51:52,765 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.124.1
:64720 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheB
us], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.124.1:64720, coord_addr=
10.40.124.1:64720, is_server=false]]}
16:51:52,828 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.124.1:6
4720, coord_addr=10.40.124.1:64720, is_server=false]
16:51:52,828 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.125.3:1739
16:51:52,875 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.125.3
:1739 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.125.3:1739, coord_addr=10
.20.110.1:2715, is_server=true]]}
16:51:52,906 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.125.3:1
739, coord_addr=10.20.110.1:2715, is_server=true]
16:51:52,906 DEBUG UDP:1542 - received (ucast) 62 bytes from /10.40.10.52:4775
16:51:52,953 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.10.52
:4775 (2 headers), size = 0 bytes], headers are {UNICAST=[UNICAST: DATA_ACK, seq
no=80], UDP=[UDP:channel_name=OSCacheBus]}
16:51:52,984 DEBUG UDP:1542 - received (ucast) 82 bytes from /10.40.10.52:4775
16:51:52,984 DEBUG UNICAST:387 - [10.20.111.1:1172] <-- ACK(10.40.10.52:4775: #8
0)
16:51:53,000 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.10.52
:4775 (3 headers), size = 0 bytes], headers are {UNICAST=[UNICAST: DATA, seqno=6
1], GMS=GmsHeader[JOIN_RSP]: join_rsp=null, UDP=[UDP:channel_name=OSCacheBus]}
16:51:53,046 DEBUG UNICAST:402 - [10.20.111.1:1172] --> ACK(10.40.10.52:4775: #6
1)
16:51:53,078 DEBUG UNICAST:344 - [10.20.111.1:1172] <-- DATA(10.40.10.52:4775: #
61, first=true
16:51:53,078 DEBUG UDP:803 - sending msg to 10.40.10.52:4775 (src=10.20.111.1:11
72), headers are {UNICAST=[UNICAST: DATA_ACK, seqno=61], UDP=[UDP:channel_name=O
SCacheBus]}
16:51:53,093  WARN ClientGmsImpl:118 - join(10.20.111.1:1172) failed, retrying
16:51:54,812 DEBUG UDP:247 - received (mcast) 51 bytes from /10.40.124.1:64678 (
size=51 bytes)
16:51:54,828 DEBUG UDP:757 - message is [dst: 231.12.21.132:45566, src: 10.40.12
4.1:64677 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCac
heBus], PING=[PING: type=GET_MBRS_REQ, arg=null]}
16:51:54,859 DEBUG PING:195 - received GET_MBRS_REQ from 10.40.124.1:64677, send
ing response [PING: type=GET_MBRS_RSP, arg=[own_addr=10.20.111.1:1172, coord_add
r=10.20.111.1:1172, is_server=false]]
16:51:54,890 DEBUG UDP:247 - received (mcast) 51 bytes from /10.20.111.1:1118 (s
ize=51 bytes)
16:51:54,890 DEBUG UDP:803 - sending msg to 10.40.124.1:64677 (src=10.20.111.1:1
172), headers are {PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.20.111.1:1172
, coord_addr=10.20.111.1:1172, is_server=false]], UDP=[UDP:channel_name=OSCacheB
us]}
16:51:54,921 DEBUG UDP:757 - message is [dst: 231.12.21.132:45566, src: 10.20.11
1.1:1117 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCach
eBus], PING=[PING: type=GET_MBRS_REQ, arg=null]}
16:51:55,000 DEBUG UDP:247 - received (mcast) 420 bytes from /10.40.121.2:1404 (
size=420 bytes)
16:51:55,000 DEBUG PING:195 - received GET_MBRS_REQ from 10.20.111.1:1117, sendi
ng response [PING: type=GET_MBRS_RSP, arg=[own_addr=10.20.111.1:1172, coord_addr
=10.20.111.1:1172, is_server=false]]
16:51:55,015 DEBUG UDP:757 - message is [dst: 231.12.21.132:45566, src: 10.40.12
1.2:1403 (2 headers), size = 355 bytes], headers are {NAKACK=[NAKACK: MSG, seqno
=632, range=null], UDP=[UDP:channel_name=OSCacheBus]}
16:51:55,046 DEBUG UDP:803 - sending msg to 10.20.111.1:1117 (src=10.20.111.1:11
72), headers are {PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.20.111.1:1172,
 coord_addr=10.20.111.1:1172, is_server=false]], UDP=[UDP:channel_name=OSCacheBu
s]}
16:51:55,078 DEBUG UDP:247 - received (mcast) 420 bytes from /10.40.121.2:1404 (
size=420 bytes)
16:51:55,078 DEBUG NAKACK:340 - message was discarded (not yet server)
16:51:55,140 DEBUG UDP:757 - message is [dst: 231.12.21.132:45566, src: 10.40.12
1.2:1403 (2 headers), size = 355 bytes], headers are {NAKACK=[NAKACK: MSG, seqno
=633, range=null], UDP=[UDP:channel_name=OSCacheBus]}
16:51:55,156 DEBUG PingWaiter:95 - clearing old responses: [[own_addr=10.20.111.
1:1117, coord_addr=10.20.111.1:1117, is_server=false], [own_addr=10.20.111.3:651
80, coord_addr=10.40.10.52:4775, is_server=true], [own_addr=10.20.111.2:1620, co
ord_addr=10.20.110.1:2715, is_server=true], [own_addr=10.20.110.3:45282, coord_a
ddr=10.40.10.52:4775, is_server=true], [own_addr=10.20.110.2:56053, coord_addr=1
0.20.110.1:2715, is_server=true], [own_addr=10.40.10.52:4775, coord_addr=10.20.1
10.1:2715, is_server=true], [own_addr=10.40.10.51:3426, coord_addr=10.20.110.1:2
715, is_server=true], [own_addr=10.40.126.2:4228, coord_addr=10.20.110.1:2715, i
s_server=true], [own_addr=10.40.120.3:4838, coord_addr=10.20.110.1:2715, is_serv
er=true], [own_addr=10.40.126.1:58260, coord_addr=10.20.110.1:2715, is_server=tr
ue], [own_addr=10.40.120.4:3696, coord_addr=10.20.110.1:2715, is_server=true], [
own_addr=10.40.121.1:3810, coord_addr=10.20.110.1:2715, is_server=true], [own_ad
dr=10.40.121.3:4249, coord_addr=10.20.110.1:2715, is_server=true], [own_addr=10.
40.124.1:64677, coord_addr=10.40.124.1:64677, is_server=false], [own_addr=10.40.
125.1:3612, coord_addr=10.20.110.1:2715, is_server=true], [own_addr=10.40.124.4:
1764, coord_addr=10.20.110.1:2715, is_server=true], [own_addr=10.40.126.3:4204,
coord_addr=10.20.110.1:2715, is_server=true], [own_addr=10.40.121.2:1403, coord_
addr=10.20.110.1:2715, is_server=true], [own_addr=10.40.121.4:1832, coord_addr=1
0.20.110.1:2715, is_server=true], [own_addr=10.40.120.2:2013, coord_addr=10.20.1
10.1:2715, is_server=true], [own_addr=10.40.124.3:15988, coord_addr=10.20.110.1:
2715, is_server=true], [own_addr=10.40.125.4:2799, coord_addr=10.20.110.1:2715,
is_server=true], [own_addr=10.40.125.2:3377, coord_addr=10.20.110.1:2715, is_ser
ver=true], [own_addr=10.40.124.2:31634, coord_addr=10.20.110.1:2715, is_server=t
rue], [own_addr=10.40.126.4:21141, coord_addr=10.20.110.1:2715, is_server=true],
 [own_addr=10.40.124.1:64720, coord_addr=10.40.124.1:64720, is_server=false], [o
wn_addr=10.40.125.3:1739, coord_addr=10.20.110.1:2715, is_server=true]]
16:51:55,187 DEBUG UDP:247 - received (mcast) 51 bytes from /10.40.124.1:64721 (
size=51 bytes)
16:51:55,187 DEBUG NAKACK:340 - message was discarded (not yet server)
16:51:55,468 DEBUG PingWaiter:107 - waiting for initial members: time_to_wait=20
00, got 0 rsps
16:51:55,484 DEBUG PingSender:57 - sending GET_MBRS_REQ
16:51:55,500 DEBUG UDP:757 - message is [dst: 231.12.21.132:45566, src: 10.40.12
4.1:64720 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCac
heBus], PING=[PING: type=GET_MBRS_REQ, arg=null]}
16:51:55,546 DEBUG UDP:803 - sending msg to 231.12.21.132:45566 (src=10.20.111.1
:1172), headers are {PING=[PING: type=GET_MBRS_REQ, arg=null], UDP=[UDP:channel_
name=OSCacheBus]}
16:51:55,578 DEBUG PING:195 - received GET_MBRS_REQ from 10.40.124.1:64720, send
ing response [PING: type=GET_MBRS_RSP, arg=[own_addr=10.20.111.1:1172, coord_add
r=10.20.111.1:1172, is_server=false]]
16:51:55,609 DEBUG UDP:820 - looped back local message [dst: 231.12.21.132:45566
, src: 10.20.111.1:1172 (2 headers), size = 0 bytes]
16:51:55,671 DEBUG PING:183 - discarded my own discovery request
16:51:55,671 DEBUG UDP:247 - received (mcast) 51 bytes from /10.20.111.1:1173 (s
ize=51 bytes)
16:51:55,671 DEBUG UDP:803 - sending msg to 10.40.124.1:64720 (src=10.20.111.1:1
172), headers are {PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.20.111.1:1172
, coord_addr=10.20.111.1:1172, is_server=false]], UDP=[UDP:channel_name=OSCacheB
us]}
16:51:55,671 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.20.111.1:1117
16:51:55,703 DEBUG UDP:748 - discarded own loopback multicast packet
16:51:55,750 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.20.111.1
:1117 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.20.111.1:1117, coord_addr=10
.20.111.1:1117, is_server=false]]}
16:51:55,812 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.20.111.1:1
117, coord_addr=10.20.111.1:1117, is_server=false]
16:51:55,812 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.20.111.3:65180
16:51:55,828 DEBUG PingWaiter:107 - waiting for initial members: time_to_wait=16
40, got 1 rsps
16:51:55,843 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.20.111.3
:65180 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheB
us], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.20.111.3:65180, coord_addr=
10.40.10.52:4775, is_server=true]]}
16:51:55,906 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.20.111.3:6
5180, coord_addr=10.40.10.52:4775, is_server=true]
16:51:55,906 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.20.110.2:56053
16:51:55,937 DEBUG PingWaiter:107 - waiting for initial members: time_to_wait=15
31, got 2 rsps
16:51:55,937 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.20.110.2
:56053 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheB
us], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.20.110.2:56053, coord_addr=
10.20.110.1:2715, is_server=true]]}
16:51:56,015 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.20.110.3:45282
16:51:56,015 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.20.110.2:5
6053, coord_addr=10.20.110.1:2715, is_server=true]
16:51:56,015 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.20.110.3
:45282 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheB
us], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.20.110.3:45282, coord_addr=
10.40.10.52:4775, is_server=true]]}
16:51:56,046 DEBUG PingWaiter:122 - initial mbrs are [[own_addr=10.20.111.1:1117
, coord_addr=10.20.111.1:1117, is_server=false], [own_addr=10.20.111.3:65180, co
ord_addr=10.40.10.52:4775, is_server=true], [own_addr=10.20.110.2:56053, coord_a
ddr=10.20.110.1:2715, is_server=true]]
16:51:56,078 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.20.111.2:1620
16:51:56,078 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.20.110.3:4
5282, coord_addr=10.40.10.52:4775, is_server=true]
16:51:56,125 DEBUG ClientGmsImpl:66 - initial_mbrs are [[own_addr=10.20.111.1:11
17, coord_addr=10.20.111.1:1117, is_server=false], [own_addr=10.20.111.3:65180,
coord_addr=10.40.10.52:4775, is_server=true], [own_addr=10.20.110.2:56053, coord
_addr=10.20.110.1:2715, is_server=true]]
16:51:56,140 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.20.111.2
:1620 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.20.111.2:1620, coord_addr=10
.20.110.1:2715, is_server=true]]}
16:51:56,203  WARN ClientGmsImpl:331 - there was more than 1 candidate for coord
inator: {10.40.10.52:4775=1, 10.20.110.1:2715=1}
16:51:56,250 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.10.51:3426
16:51:56,250 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.20.111.2:1
620, coord_addr=10.20.110.1:2715, is_server=true]
16:51:56,281 DEBUG ClientGmsImpl:113 - sending handleJoin(10.20.111.1:1172) to 1
0.40.10.52:4775
16:51:56,281 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.10.51
:3426 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.10.51:3426, coord_addr=10
.20.110.1:2715, is_server=true]]}
16:51:56,328 DEBUG UNICAST:222 - [10.20.111.1:1172] --> DATA(10.40.10.52:4775: #
81, first=false)
16:51:56,375 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.124.1:64677
16:51:56,375 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.10.51:3
426, coord_addr=10.20.110.1:2715, is_server=true]
16:51:56,406 DEBUG UDP:803 - sending msg to 10.40.10.52:4775 (src=10.20.111.1:11
72), headers are {UNICAST=[UNICAST: DATA, seqno=81], GMS=GmsHeader[JOIN_REQ]: mb
r=10.20.111.1:1172, UDP=[UDP:channel_name=OSCacheBus]}
16:51:56,406 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.124.1
:64677 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheB
us], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.124.1:64677, coord_addr=
10.40.124.1:64677, is_server=false]]}
16:51:56,515 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.121.1:3810
16:51:56,515 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.124.1:6
4677, coord_addr=10.40.124.1:64677, is_server=false]
16:51:56,515 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.121.1
:3810 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.121.1:3810, coord_addr=10
.20.110.1:2715, is_server=true]]}
16:51:56,593 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.121.1:3
810, coord_addr=10.20.110.1:2715, is_server=true]
16:51:56,593 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.10.52:4775
16:51:56,625 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.10.52
:4775 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.10.52:4775, coord_addr=10
.20.110.1:2715, is_server=true]]}
16:51:56,671 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.10.52:4
775, coord_addr=10.20.110.1:2715, is_server=true]
16:51:56,671 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.124.3:15988
16:51:56,703 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.124.3
:15988 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheB
us], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.124.3:15988, coord_addr=
10.20.110.1:2715, is_server=true]]}
16:51:56,750 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.126.2:4228
16:51:56,750 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.124.3:1
5988, coord_addr=10.20.110.1:2715, is_server=true]
16:51:56,765 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.126.2
:4228 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.126.2:4228, coord_addr=10
.20.110.1:2715, is_server=true]]}
16:51:56,765 DEBUG UDP:247 - received (mcast) 420 bytes from /10.40.124.4:1765 (
size=420 bytes)
16:51:56,828 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.120.3:4838
16:51:56,828 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.126.2:4
228, coord_addr=10.20.110.1:2715, is_server=true]
16:51:56,843 DEBUG UDP:757 - message is [dst: 231.12.21.132:45566, src: 10.40.12
4.4:1764 (2 headers), size = 355 bytes], headers are {NAKACK=[NAKACK: MSG, seqno
=539, range=null], UDP=[UDP:channel_name=OSCacheBus]}
16:51:56,859 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.120.3
:4838 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.120.3:4838, coord_addr=10
.20.110.1:2715, is_server=true]]}
16:51:56,921 DEBUG UDP:247 - received (mcast) 420 bytes from /10.40.124.4:1765 (
size=420 bytes)
16:51:56,921 DEBUG NAKACK:340 - message was discarded (not yet server)
16:51:56,953 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.125.4:2799
16:51:56,953 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.120.3:4
838, coord_addr=10.20.110.1:2715, is_server=true]
16:51:56,984 DEBUG UDP:757 - message is [dst: 231.12.21.132:45566, src: 10.40.12
4.4:1764 (2 headers), size = 355 bytes], headers are {NAKACK=[NAKACK: MSG, seqno
=540, range=null], UDP=[UDP:channel_name=OSCacheBus]}
16:51:57,000 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.125.4
:2799 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.125.4:2799, coord_addr=10
.20.110.1:2715, is_server=true]]}
16:51:57,062 DEBUG NAKACK:340 - message was discarded (not yet server)
16:51:57,109 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.126.1:58260
16:51:57,109 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.125.4:2
799, coord_addr=10.20.110.1:2715, is_server=true]
16:51:57,125 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.126.1
:58260 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheB
us], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.126.1:58260, coord_addr=
10.20.110.1:2715, is_server=true]]}
16:51:57,203 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.126.1:5
8260, coord_addr=10.20.110.1:2715, is_server=true]
16:51:57,203 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.121.4:1832
16:51:57,234 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.121.4
:1832 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.121.4:1832, coord_addr=10
.20.110.1:2715, is_server=true]]}
16:51:57,281 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.121.2:1403
16:51:57,281 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.121.4:1
832, coord_addr=10.20.110.1:2715, is_server=true]
16:51:57,296 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.121.2
:1403 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.121.2:1403, coord_addr=10
.20.110.1:2715, is_server=true]]}
16:51:57,359 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.121.2:1
403, coord_addr=10.20.110.1:2715, is_server=true]
16:51:57,359 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.126.3:4204
16:51:57,390 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.126.3
:4204 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.126.3:4204, coord_addr=10
.20.110.1:2715, is_server=true]]}
16:51:57,437 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.124.4:1764
16:51:57,437 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.126.3:4
204, coord_addr=10.20.110.1:2715, is_server=true]
16:51:57,453 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.124.4
:1764 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.124.4:1764, coord_addr=10
.20.110.1:2715, is_server=true]]}
16:51:57,515 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.125.3:1739
16:51:57,515 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.124.4:1
764, coord_addr=10.20.110.1:2715, is_server=true]
16:51:57,531 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.125.3
:1739 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.125.3:1739, coord_addr=10
.20.110.1:2715, is_server=true]]}
16:51:57,593 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.125.3:1
739, coord_addr=10.20.110.1:2715, is_server=true]
16:51:57,593 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.121.3:4249
16:51:57,625 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.121.3
:4249 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.121.3:4249, coord_addr=10
.20.110.1:2715, is_server=true]]}
16:51:57,671 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.121.3:4
249, coord_addr=10.20.110.1:2715, is_server=true]
16:51:57,671 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.124.2:31634
16:51:57,703 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.124.2
:31634 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheB
us], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.124.2:31634, coord_addr=
10.20.110.1:2715, is_server=true]]}
16:51:57,750 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.124.2:3
1634, coord_addr=10.20.110.1:2715, is_server=true]
16:51:57,750 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.125.2:3377
16:51:57,781 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.125.2
:3377 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.125.2:3377, coord_addr=10
.20.110.1:2715, is_server=true]]}
16:51:57,828 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.120.4:3696
16:51:57,828 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.125.2:3
377, coord_addr=10.20.110.1:2715, is_server=true]
16:51:57,843 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.120.4
:3696 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.120.4:3696, coord_addr=10
.20.110.1:2715, is_server=true]]}
16:51:57,906 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.120.4:3
696, coord_addr=10.20.110.1:2715, is_server=true]
16:51:57,906 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.125.1:3612
16:51:57,937 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.125.1
:3612 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.125.1:3612, coord_addr=10
.20.110.1:2715, is_server=true]]}
16:51:57,984 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.120.2:2013
16:51:57,984 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.125.1:3
612, coord_addr=10.20.110.1:2715, is_server=true]
16:51:58,000 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.120.2
:2013 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.120.2:2013, coord_addr=10
.20.110.1:2715, is_server=true]]}
16:51:58,046 DEBUG UDP:247 - received (mcast) 420 bytes from /10.40.124.4:1765 (
size=420 bytes)
16:51:58,062 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.126.4:21141
16:51:58,062 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.120.2:2
013, coord_addr=10.20.110.1:2715, is_server=true]
16:51:58,078 DEBUG UDP:757 - message is [dst: 231.12.21.132:45566, src: 10.40.12
4.4:1764 (2 headers), size = 355 bytes], headers are {NAKACK=[NAKACK: MSG, seqno
=541, range=null], UDP=[UDP:channel_name=OSCacheBus]}
16:51:58,093 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.126.4
:21141 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheB
us], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.126.4:21141, coord_addr=
10.20.110.1:2715, is_server=true]]}
16:51:58,156 DEBUG UDP:247 - received (mcast) 420 bytes from /10.40.124.4:1765 (
size=420 bytes)
16:51:58,156 DEBUG NAKACK:340 - message was discarded (not yet server)
16:51:58,187 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.126.4:2
1141, coord_addr=10.20.110.1:2715, is_server=true]
16:51:58,187 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.124.1:64720
16:51:58,218 DEBUG UDP:757 - message is [dst: 231.12.21.132:45566, src: 10.40.12
4.4:1764 (2 headers), size = 355 bytes], headers are {NAKACK=[NAKACK: MSG, seqno
=542, range=null], UDP=[UDP:channel_name=OSCacheBus]}
16:51:58,265 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.124.1
:64720 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheB
us], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.124.1:64720, coord_addr=
10.40.124.1:64720, is_server=false]]}
16:51:58,296 DEBUG NAKACK:340 - message was discarded (not yet server)
16:51:58,343 DEBUG UDP:1542 - received (ucast) 62 bytes from /10.40.10.52:4775
16:51:58,343 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.124.1:6
4720, coord_addr=10.40.124.1:64720, is_server=false]
16:51:58,375 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.10.52
:4775 (2 headers), size = 0 bytes], headers are {UNICAST=[UNICAST: DATA_ACK, seq
no=81], UDP=[UDP:channel_name=OSCacheBus]}
16:51:58,406 DEBUG UDP:247 - received (mcast) 420 bytes from /10.40.121.2:1404 (
size=420 bytes)
16:51:58,421 DEBUG UDP:1542 - received (ucast) 82 bytes from /10.40.10.52:4775
16:51:58,421 DEBUG UNICAST:387 - [10.20.111.1:1172] <-- ACK(10.40.10.52:4775: #8
1)
16:51:58,453 DEBUG UDP:757 - message is [dst: 231.12.21.132:45566, src: 10.40.12
1.2:1403 (2 headers), size = 355 bytes], headers are {NAKACK=[NAKACK: MSG, seqno
=634, range=null], UDP=[UDP:channel_name=OSCacheBus]}
16:51:58,453 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.10.52
:4775 (3 headers), size = 0 bytes], headers are {UNICAST=[UNICAST: DATA, seqno=6
2], GMS=GmsHeader[JOIN_RSP]: join_rsp=null, UDP=[UDP:channel_name=OSCacheBus]}
16:51:58,515 DEBUG UDP:247 - received (mcast) 420 bytes from /10.40.121.2:1404 (
size=420 bytes)
16:51:58,515 DEBUG NAKACK:340 - message was discarded (not yet server)
16:51:58,578 DEBUG UDP:757 - message is [dst: 231.12.21.132:45566, src: 10.40.12
1.2:1403 (2 headers), size = 355 bytes], headers are {NAKACK=[NAKACK: MSG, seqno
=635, range=null], UDP=[UDP:channel_name=OSCacheBus]}
16:51:58,578 DEBUG UNICAST:402 - [10.20.111.1:1172] --> ACK(10.40.10.52:4775: #6
2)
16:51:58,625 DEBUG UDP:247 - received (mcast) 420 bytes from /10.40.121.3:4250 (
size=420 bytes)
16:51:58,625 DEBUG NAKACK:340 - message was discarded (not yet server)
16:51:58,640 DEBUG UNICAST:344 - [10.20.111.1:1172] <-- DATA(10.40.10.52:4775: #
62, first=false
16:51:58,640 DEBUG UDP:803 - sending msg to 10.40.10.52:4775 (src=10.20.111.1:11
72), headers are {UNICAST=[UNICAST: DATA_ACK, seqno=62], UDP=[UDP:channel_name=O
SCacheBus]}
16:51:58,656 DEBUG UDP:757 - message is [dst: 231.12.21.132:45566, src: 10.40.12
1.3:4249 (2 headers), size = 355 bytes], headers are {NAKACK=[NAKACK: MSG, seqno
=599, range=null], UDP=[UDP:channel_name=OSCacheBus]}
16:51:58,703  WARN ClientGmsImpl:118 - join(10.20.111.1:1172) failed, retrying
16:51:58,765 DEBUG UDP:247 - received (mcast) 420 bytes from /10.40.121.3:4250 (
size=420 bytes)
16:51:58,765 DEBUG NAKACK:340 - message was discarded (not yet server)
16:51:58,796 DEBUG UDP:757 - message is [dst: 231.12.21.132:45566, src: 10.40.12
1.3:4249 (2 headers), size = 355 bytes], headers are {NAKACK=[NAKACK: MSG, seqno
=600, range=null], UDP=[UDP:channel_name=OSCacheBus]}
16:51:58,843 DEBUG UDP:247 - received (mcast) 420 bytes from /10.40.124.4:1765 (
size=420 bytes)
16:51:58,843 DEBUG NAKACK:340 - message was discarded (not yet server)
16:51:58,875 DEBUG UDP:757 - message is [dst: 231.12.21.132:45566, src: 10.40.12
4.4:1764 (2 headers), size = 355 bytes], headers are {NAKACK=[NAKACK: MSG, seqno
=543, range=null], UDP=[UDP:channel_name=OSCacheBus]}
16:51:58,906 DEBUG UDP:247 - received (mcast) 420 bytes from /10.40.124.4:1765 (
size=420 bytes)
16:51:58,906 DEBUG NAKACK:340 - message was discarded (not yet server)
16:51:58,937 DEBUG UDP:757 - message is [dst: 231.12.21.132:45566, src: 10.40.12
4.4:1764 (2 headers), size = 355 bytes], headers are {NAKACK=[NAKACK: MSG, seqno
=544, range=null], UDP=[UDP:channel_name=OSCacheBus]}
16:51:58,984 DEBUG NAKACK:340 - message was discarded (not yet server)
16:52:00,734 DEBUG UDP:247 - received (mcast) 420 bytes from /10.40.125.1:3613 (
size=420 bytes)
16:52:00,750 DEBUG UDP:757 - message is [dst: 231.12.21.132:45566, src: 10.40.12
5.1:3612 (2 headers), size = 355 bytes], headers are {NAKACK=[NAKACK: MSG, seqno
=805, range=null], UDP=[UDP:channel_name=OSCacheBus]}
16:52:00,781 DEBUG UDP:247 - received (mcast) 420 bytes from /10.40.125.1:3613 (
size=420 bytes)
16:52:00,781 DEBUG NAKACK:340 - message was discarded (not yet server)
16:52:00,796 DEBUG PingWaiter:95 - clearing old responses: [[own_addr=10.20.111.
1:1117, coord_addr=10.20.111.1:1117, is_server=false], [own_addr=10.20.111.3:651
80, coord_addr=10.40.10.52:4775, is_server=true], [own_addr=10.20.110.2:56053, c
oord_addr=10.20.110.1:2715, is_server=true], [own_addr=10.20.110.3:45282, coord_
addr=10.40.10.52:4775, is_server=true], [own_addr=10.20.111.2:1620, coord_addr=1
0.20.110.1:2715, is_server=true], [own_addr=10.40.10.51:3426, coord_addr=10.20.1
10.1:2715, is_server=true], [own_addr=10.40.124.1:64677, coord_addr=10.40.124.1:
64677, is_server=false], [own_addr=10.40.121.1:3810, coord_addr=10.20.110.1:2715
, is_server=true], [own_addr=10.40.10.52:4775, coord_addr=10.20.110.1:2715, is_s
erver=true], [own_addr=10.40.124.3:15988, coord_addr=10.20.110.1:2715, is_server
=true], [own_addr=10.40.126.2:4228, coord_addr=10.20.110.1:2715, is_server=true]
, [own_addr=10.40.120.3:4838, coord_addr=10.20.110.1:2715, is_server=true], [own
_addr=10.40.125.4:2799, coord_addr=10.20.110.1:2715, is_server=true], [own_addr=
10.40.126.1:58260, coord_addr=10.20.110.1:2715, is_server=true], [own_addr=10.40
.121.4:1832, coord_addr=10.20.110.1:2715, is_server=true], [own_addr=10.40.121.2
:1403, coord_addr=10.20.110.1:2715, is_server=true], [own_addr=10.40.126.3:4204,
 coord_addr=10.20.110.1:2715, is_server=true], [own_addr=10.40.124.4:1764, coord
_addr=10.20.110.1:2715, is_server=true], [own_addr=10.40.125.3:1739, coord_addr=
10.20.110.1:2715, is_server=true], [own_addr=10.40.121.3:4249, coord_addr=10.20.
110.1:2715, is_server=true], [own_addr=10.40.124.2:31634, coord_addr=10.20.110.1
:2715, is_server=true], [own_addr=10.40.125.2:3377, coord_addr=10.20.110.1:2715,
 is_server=true], [own_addr=10.40.120.4:3696, coord_addr=10.20.110.1:2715, is_se
rver=true], [own_addr=10.40.125.1:3612, coord_addr=10.20.110.1:2715, is_server=t
rue], [own_addr=10.40.120.2:2013, coord_addr=10.20.110.1:2715, is_server=true],
[own_addr=10.40.126.4:21141, coord_addr=10.20.110.1:2715, is_server=true], [own_
addr=10.40.124.1:64720, coord_addr=10.40.124.1:64720, is_server=false]]
16:52:00,812 DEBUG UDP:757 - message is [dst: 231.12.21.132:45566, src: 10.40.12
5.1:3612 (2 headers), size = 355 bytes], headers are {NAKACK=[NAKACK: MSG, seqno
=806, range=null], UDP=[UDP:channel_name=OSCacheBus]}
16:52:01,093 DEBUG PingWaiter:107 - waiting for initial members: time_to_wait=20
00, got 0 rsps
16:52:01,093 DEBUG PingSender:57 - sending GET_MBRS_REQ
16:52:01,140 DEBUG NAKACK:340 - message was discarded (not yet server)
16:52:01,171 DEBUG UDP:803 - sending msg to 231.12.21.132:45566 (src=10.20.111.1
:1172), headers are {PING=[PING: type=GET_MBRS_REQ, arg=null], UDP=[UDP:channel_
name=OSCacheBus]}
16:52:01,218 DEBUG UDP:820 - looped back local message [dst: 231.12.21.132:45566
, src: 10.20.111.1:1172 (2 headers), size = 0 bytes]
16:52:01,234 DEBUG PING:183 - discarded my own discovery request
16:52:01,234 DEBUG UDP:247 - received (mcast) 51 bytes from /10.20.111.1:1173 (s
ize=51 bytes)
16:52:01,234 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.20.111.1:1117
16:52:01,265 DEBUG UDP:748 - discarded own loopback multicast packet
16:52:01,281 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.20.111.1
:1117 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.20.111.1:1117, coord_addr=10
.20.111.1:1117, is_server=false]]}
16:52:01,343 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.20.111.3:65180
16:52:01,343 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.20.111.1:1
117, coord_addr=10.20.111.1:1117, is_server=false]
16:52:01,359 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.20.111.3
:65180 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheB
us], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.20.111.3:65180, coord_addr=
10.40.10.52:4775, is_server=true]]}
16:52:01,375 DEBUG PingWaiter:107 - waiting for initial members: time_to_wait=17
18, got 1 rsps
16:52:01,421 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.20.110.2:56053
16:52:01,421 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.20.111.3:6
5180, coord_addr=10.40.10.52:4775, is_server=true]
16:52:01,453 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.20.110.2
:56053 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheB
us], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.20.110.2:56053, coord_addr=
10.20.110.1:2715, is_server=true]]}
16:52:01,484 DEBUG PingWaiter:107 - waiting for initial members: time_to_wait=16
09, got 2 rsps
16:52:01,531 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.20.110.2:5
6053, coord_addr=10.20.110.1:2715, is_server=true]
16:52:01,531 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.20.110.3:45282
16:52:01,562 DEBUG PingWaiter:122 - initial mbrs are [[own_addr=10.20.111.1:1117
, coord_addr=10.20.111.1:1117, is_server=false], [own_addr=10.20.111.3:65180, co
ord_addr=10.40.10.52:4775, is_server=true], [own_addr=10.20.110.2:56053, coord_a
ddr=10.20.110.1:2715, is_server=true]]
16:52:01,578 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.20.110.3
:45282 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheB
us], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.20.110.3:45282, coord_addr=
10.40.10.52:4775, is_server=true]]}
16:52:01,625 DEBUG ClientGmsImpl:66 - initial_mbrs are [[own_addr=10.20.111.1:11
17, coord_addr=10.20.111.1:1117, is_server=false], [own_addr=10.20.111.3:65180,
coord_addr=10.40.10.52:4775, is_server=true], [own_addr=10.20.110.2:56053, coord
_addr=10.20.110.1:2715, is_server=true]]
16:52:01,671 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.20.111.2:1620
16:52:01,671 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.20.110.3:4
5282, coord_addr=10.40.10.52:4775, is_server=true]
16:52:01,718  WARN ClientGmsImpl:331 - there was more than 1 candidate for coord
inator: {10.40.10.52:4775=1, 10.20.110.1:2715=1}
16:52:01,718 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.20.111.2
:1620 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.20.111.2:1620, coord_addr=10
.20.110.1:2715, is_server=true]]}
16:52:01,765 DEBUG ClientGmsImpl:113 - sending handleJoin(10.20.111.1:1172) to 1
0.40.10.52:4775
16:52:01,812 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.10.51:3426
16:52:01,812 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.20.111.2:1
620, coord_addr=10.20.110.1:2715, is_server=true]
16:52:01,812 DEBUG UDP:247 - received (mcast) 51 bytes from /10.40.124.1:64678 (
size=51 bytes)
16:52:01,828 DEBUG UNICAST:222 - [10.20.111.1:1172] --> DATA(10.40.10.52:4775: #
82, first=false)
16:52:01,843 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.10.51
:3426 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.10.51:3426, coord_addr=10
.20.110.1:2715, is_server=true]]}
16:52:01,890 DEBUG UDP:757 - message is [dst: 231.12.21.132:45566, src: 10.40.12
4.1:64677 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCac
heBus], PING=[PING: type=GET_MBRS_REQ, arg=null]}
16:52:01,906 DEBUG UDP:803 - sending msg to 10.40.10.52:4775 (src=10.20.111.1:11
72), headers are {UNICAST=[UNICAST: DATA, seqno=82], GMS=GmsHeader[JOIN_REQ]: mb
r=10.20.111.1:1172, UDP=[UDP:channel_name=OSCacheBus]}
16:52:01,953 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.10.52:4775
16:52:01,953 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.10.51:3
426, coord_addr=10.20.110.1:2715, is_server=true]
16:52:01,984 DEBUG UDP:247 - received (mcast) 51 bytes from /10.20.111.1:1118 (s
ize=51 bytes)
16:52:02,031 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.10.52
:4775 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.10.52:4775, coord_addr=10
.20.110.1:2715, is_server=true]]}
16:52:02,062 DEBUG PING:195 - received GET_MBRS_REQ from 10.40.124.1:64677, send
ing response [PING: type=GET_MBRS_RSP, arg=[own_addr=10.20.111.1:1172, coord_add
r=10.20.111.1:1172, is_server=false]]
16:52:02,078 DEBUG UDP:757 - message is [dst: 231.12.21.132:45566, src: 10.20.11
1.1:1117 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCach
eBus], PING=[PING: type=GET_MBRS_REQ, arg=null]}
16:52:02,125 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.121.3:4249
16:52:02,156 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.10.52:4
775, coord_addr=10.20.110.1:2715, is_server=true]
16:52:02,156 DEBUG UDP:803 - sending msg to 10.40.124.1:64677 (src=10.20.111.1:1
172), headers are {PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.20.111.1:1172
, coord_addr=10.20.111.1:1172, is_server=false]], UDP=[UDP:channel_name=OSCacheB
us]}
16:52:02,187 DEBUG UDP:247 - received (mcast) 51 bytes from /10.40.124.1:64721 (
size=51 bytes)
16:52:02,203 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.121.3
:4249 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.121.3:4249, coord_addr=10
.20.110.1:2715, is_server=true]]}
16:52:02,234 DEBUG PING:195 - received GET_MBRS_REQ from 10.20.111.1:1117, sendi
ng response [PING: type=GET_MBRS_RSP, arg=[own_addr=10.20.111.1:1172, coord_addr
=10.20.111.1:1172, is_server=false]]
16:52:02,296 DEBUG UDP:757 - message is [dst: 231.12.21.132:45566, src: 10.40.12
4.1:64720 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCac
heBus], PING=[PING: type=GET_MBRS_REQ, arg=null]}
16:52:02,343 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.125.3:1739
16:52:02,375 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.121.3:4
249, coord_addr=10.20.110.1:2715, is_server=true]
16:52:02,375 DEBUG UDP:803 - sending msg to 10.20.111.1:1117 (src=10.20.111.1:11
72), headers are {PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.20.111.1:1172,
 coord_addr=10.20.111.1:1172, is_server=false]], UDP=[UDP:channel_name=OSCacheBu
s]}
16:52:02,421 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.125.3
:1739 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.125.3:1739, coord_addr=10
.20.110.1:2715, is_server=true]]}
16:52:02,437 DEBUG PING:195 - received GET_MBRS_REQ from 10.40.124.1:64720, send
ing response [PING: type=GET_MBRS_RSP, arg=[own_addr=10.20.111.1:1172, coord_add
r=10.20.111.1:1172, is_server=false]]
16:52:02,531 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.126.2:4228
16:52:02,562 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.125.3:1
739, coord_addr=10.20.110.1:2715, is_server=true]
16:52:02,562 DEBUG UDP:803 - sending msg to 10.40.124.1:64720 (src=10.20.111.1:1
172), headers are {PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.20.111.1:1172
, coord_addr=10.20.111.1:1172, is_server=false]], UDP=[UDP:channel_name=OSCacheB
us]}
16:52:02,578 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.126.2
:4228 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.126.2:4228, coord_addr=10
.20.110.1:2715, is_server=true]]}
16:52:02,687 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.126.2:4
228, coord_addr=10.20.110.1:2715, is_server=true]
16:52:02,687 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.124.1:64677
16:52:02,718 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.124.1
:64677 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheB
us], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.124.1:64677, coord_addr=
10.40.124.1:64677, is_server=false]]}
16:52:02,765 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.124.2:31634
16:52:02,765 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.124.1:6
4677, coord_addr=10.40.124.1:64677, is_server=false]
16:52:02,781 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.124.2
:31634 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheB
us], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.124.2:31634, coord_addr=
10.20.110.1:2715, is_server=true]]}
16:52:02,843 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.124.2:3
1634, coord_addr=10.20.110.1:2715, is_server=true]
16:52:02,843 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.121.2:1403
16:52:02,875 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.121.2
:1403 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.121.2:1403, coord_addr=10
.20.110.1:2715, is_server=true]]}
16:52:02,906 DEBUG UDP:247 - received (mcast) 420 bytes from /10.40.121.3:4250 (
size=420 bytes)
16:52:02,921 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.120.3:4838
16:52:02,921 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.121.2:1
403, coord_addr=10.20.110.1:2715, is_server=true]
16:52:02,937 DEBUG UDP:757 - message is [dst: 231.12.21.132:45566, src: 10.40.12
1.3:4249 (2 headers), size = 355 bytes], headers are {NAKACK=[NAKACK: MSG, seqno
=601, range=null], UDP=[UDP:channel_name=OSCacheBus]}
16:52:02,953 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.120.3
:4838 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.120.3:4838, coord_addr=10
.20.110.1:2715, is_server=true]]}
16:52:03,015 DEBUG UDP:247 - received (mcast) 420 bytes from /10.40.121.3:4250 (
size=420 bytes)
16:52:03,015 DEBUG NAKACK:340 - message was discarded (not yet server)
16:52:03,046 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.126.3:4204
16:52:03,046 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.120.3:4
838, coord_addr=10.20.110.1:2715, is_server=true]
16:52:03,078 DEBUG UDP:757 - message is [dst: 231.12.21.132:45566, src: 10.40.12
1.3:4249 (2 headers), size = 355 bytes], headers are {NAKACK=[NAKACK: MSG, seqno
=602, range=null], UDP=[UDP:channel_name=OSCacheBus]}
16:52:03,093 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.126.3
:4204 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.126.3:4204, coord_addr=10
.20.110.1:2715, is_server=true]]}
16:52:03,156 DEBUG NAKACK:340 - message was discarded (not yet server)
16:52:03,203 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.126.1:58260
16:52:03,203 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.126.3:4
204, coord_addr=10.20.110.1:2715, is_server=true]
16:52:03,218 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.126.1
:58260 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheB
us], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.126.1:58260, coord_addr=
10.20.110.1:2715, is_server=true]]}
16:52:03,296 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.126.1:5
8260, coord_addr=10.20.110.1:2715, is_server=true]
16:52:03,296 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.125.2:3377
16:52:03,328 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.125.2
:3377 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.125.2:3377, coord_addr=10
.20.110.1:2715, is_server=true]]}
16:52:03,375 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.124.4:1764
16:52:03,375 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.125.2:3
377, coord_addr=10.20.110.1:2715, is_server=true]
16:52:03,390 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.124.4
:1764 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.124.4:1764, coord_addr=10
.20.110.1:2715, is_server=true]]}
16:52:03,453 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.121.4:1832
16:52:03,453 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.124.4:1
764, coord_addr=10.20.110.1:2715, is_server=true]
16:52:03,468 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.121.4
:1832 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.121.4:1832, coord_addr=10
.20.110.1:2715, is_server=true]]}
16:52:03,531 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.121.4:1
832, coord_addr=10.20.110.1:2715, is_server=true]
16:52:03,531 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.125.4:2799
16:52:03,562 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.125.4
:2799 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.125.4:2799, coord_addr=10
.20.110.1:2715, is_server=true]]}
16:52:03,609 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.125.4:2
799, coord_addr=10.20.110.1:2715, is_server=true]
16:52:03,609 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.120.4:3696
16:52:03,640 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.120.4
:3696 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.120.4:3696, coord_addr=10
.20.110.1:2715, is_server=true]]}
16:52:03,687 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.120.4:3
696, coord_addr=10.20.110.1:2715, is_server=true]
16:52:03,687 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.120.2:2013
16:52:03,718 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.120.2
:2013 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.120.2:2013, coord_addr=10
.20.110.1:2715, is_server=true]]}
16:52:03,765 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.126.4:21141
16:52:03,765 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.120.2:2
013, coord_addr=10.20.110.1:2715, is_server=true]
16:52:03,781 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.126.4
:21141 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheB
us], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.126.4:21141, coord_addr=
10.20.110.1:2715, is_server=true]]}
16:52:03,843 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.126.4:2
1141, coord_addr=10.20.110.1:2715, is_server=true]
16:52:03,843 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.124.3:15988
16:52:03,875 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.124.3
:15988 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheB
us], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.124.3:15988, coord_addr=
10.20.110.1:2715, is_server=true]]}
16:52:03,921 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.125.1:3612
16:52:03,921 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.124.3:1
5988, coord_addr=10.20.110.1:2715, is_server=true]
16:52:03,937 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.125.1
:3612 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.125.1:3612, coord_addr=10
.20.110.1:2715, is_server=true]]}
16:52:04,000 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.125.1:3
612, coord_addr=10.20.110.1:2715, is_server=true]
16:52:04,000 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.124.1:64720
16:52:04,031 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.124.1
:64720 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheB
us], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.124.1:64720, coord_addr=
10.40.124.1:64720, is_server=false]]}
16:52:04,078 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.121.1:3810
16:52:04,078 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.124.1:6
4720, coord_addr=10.40.124.1:64720, is_server=false]
16:52:04,093 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.121.1
:3810 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.121.1:3810, coord_addr=10
.20.110.1:2715, is_server=true]]}
16:52:04,156 DEBUG UDP:1542 - received (ucast) 62 bytes from /10.40.10.52:4775
16:52:04,156 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.121.1:3
810, coord_addr=10.20.110.1:2715, is_server=true]
16:52:04,171 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.10.52
:4775 (2 headers), size = 0 bytes], headers are {UNICAST=[UNICAST: DATA_ACK, seq
no=82], UDP=[UDP:channel_name=OSCacheBus]}
16:52:04,218 DEBUG UDP:1542 - received (ucast) 82 bytes from /10.40.10.52:4775
16:52:04,218 DEBUG UNICAST:387 - [10.20.111.1:1172] <-- ACK(10.40.10.52:4775: #8
2)
16:52:04,234 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.10.52
:4775 (3 headers), size = 0 bytes], headers are {UNICAST=[UNICAST: DATA, seqno=6
3], GMS=GmsHeader[JOIN_RSP]: join_rsp=null, UDP=[UDP:channel_name=OSCacheBus]}
16:52:04,281 DEBUG UNICAST:402 - [10.20.111.1:1172] --> ACK(10.40.10.52:4775: #6
3)
16:52:04,312 DEBUG UNICAST:344 - [10.20.111.1:1172] <-- DATA(10.40.10.52:4775: #
63, first=false
16:52:04,312 DEBUG UDP:803 - sending msg to 10.40.10.52:4775 (src=10.20.111.1:11
72), headers are {UNICAST=[UNICAST: DATA_ACK, seqno=63], UDP=[UDP:channel_name=O
SCacheBus]}
16:52:04,328  WARN ClientGmsImpl:118 - join(10.20.111.1:1172) failed, retrying
16:52:05,312 DEBUG UDP:247 - received (mcast) 420 bytes from /10.40.121.3:4250 (
size=420 bytes)
16:52:05,343 DEBUG UDP:757 - message is [dst: 231.12.21.132:45566, src: 10.40.12
1.3:4249 (2 headers), size = 355 bytes], headers are {NAKACK=[NAKACK: MSG, seqno
=603, range=null], UDP=[UDP:channel_name=OSCacheBus]}
16:52:05,375 DEBUG UDP:247 - received (mcast) 420 bytes from /10.40.121.3:4250 (
size=420 bytes)
16:52:05,375 DEBUG NAKACK:340 - message was discarded (not yet server)
16:52:05,390 DEBUG UDP:757 - message is [dst: 231.12.21.132:45566, src: 10.40.12
1.3:4249 (2 headers), size = 355 bytes], headers are {NAKACK=[NAKACK: MSG, seqno
=604, range=null], UDP=[UDP:channel_name=OSCacheBus]}
16:52:05,437 DEBUG NAKACK:340 - message was discarded (not yet server)
16:52:06,375 DEBUG PingWaiter:95 - clearing old responses: [[own_addr=10.20.111.
1:1117, coord_addr=10.20.111.1:1117, is_server=false], [own_addr=10.20.111.3:651
80, coord_addr=10.40.10.52:4775, is_server=true], [own_addr=10.20.110.2:56053, c
oord_addr=10.20.110.1:2715, is_server=true], [own_addr=10.20.110.3:45282, coord_
addr=10.40.10.52:4775, is_server=true], [own_addr=10.20.111.2:1620, coord_addr=1
0.20.110.1:2715, is_server=true], [own_addr=10.40.10.51:3426, coord_addr=10.20.1
10.1:2715, is_server=true], [own_addr=10.40.10.52:4775, coord_addr=10.20.110.1:2
715, is_server=true], [own_addr=10.40.121.3:4249, coord_addr=10.20.110.1:2715, i
s_server=true], [own_addr=10.40.125.3:1739, coord_addr=10.20.110.1:2715, is_serv
er=true], [own_addr=10.40.126.2:4228, coord_addr=10.20.110.1:2715, is_server=tru
e], [own_addr=10.40.124.1:64677, coord_addr=10.40.124.1:64677, is_server=false],
 [own_addr=10.40.124.2:31634, coord_addr=10.20.110.1:2715, is_server=true], [own
_addr=10.40.121.2:1403, coord_addr=10.20.110.1:2715, is_server=true], [own_addr=
10.40.120.3:4838, coord_addr=10.20.110.1:2715, is_server=true], [own_addr=10.40.
126.3:4204, coord_addr=10.20.110.1:2715, is_server=true], [own_addr=10.40.126.1:
58260, coord_addr=10.20.110.1:2715, is_server=true], [own_addr=10.40.125.2:3377,
 coord_addr=10.20.110.1:2715, is_server=true], [own_addr=10.40.124.4:1764, coord
_addr=10.20.110.1:2715, is_server=true], [own_addr=10.40.121.4:1832, coord_addr=
10.20.110.1:2715, is_server=true], [own_addr=10.40.125.4:2799, coord_addr=10.20.
110.1:2715, is_server=true], [own_addr=10.40.120.4:3696, coord_addr=10.20.110.1:
2715, is_server=true], [own_addr=10.40.120.2:2013, coord_addr=10.20.110.1:2715,
is_server=true], [own_addr=10.40.126.4:21141, coord_addr=10.20.110.1:2715, is_se
rver=true], [own_addr=10.40.124.3:15988, coord_addr=10.20.110.1:2715, is_server=
true], [own_addr=10.40.125.1:3612, coord_addr=10.20.110.1:2715, is_server=true],
 [own_addr=10.40.124.1:64720, coord_addr=10.40.124.1:64720, is_server=false], [o
wn_addr=10.40.121.1:3810, coord_addr=10.20.110.1:2715, is_server=true]]
16:52:06,640 DEBUG PingWaiter:107 - waiting for initial members: time_to_wait=20
00, got 0 rsps
16:52:06,640 DEBUG PingSender:57 - sending GET_MBRS_REQ
16:52:06,671 DEBUG UDP:803 - sending msg to 231.12.21.132:45566 (src=10.20.111.1
:1172), headers are {PING=[PING: type=GET_MBRS_REQ, arg=null], UDP=[UDP:channel_
name=OSCacheBus]}
16:52:06,703 DEBUG UDP:820 - looped back local message [dst: 231.12.21.132:45566
, src: 10.20.111.1:1172 (2 headers), size = 0 bytes]
16:52:06,734 DEBUG PING:183 - discarded my own discovery request
16:52:06,734 DEBUG UDP:247 - received (mcast) 51 bytes from /10.20.111.1:1173 (s
ize=51 bytes)
16:52:06,734 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.20.111.1:1117
16:52:06,765 DEBUG UDP:748 - discarded own loopback multicast packet
16:52:06,781 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.20.111.1
:1117 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.20.111.1:1117, coord_addr=10
.20.111.1:1117, is_server=false]]}
16:52:06,828 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.20.111.3:65180
16:52:06,828 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.20.111.1:1
117, coord_addr=10.20.111.1:1117, is_server=false]
16:52:06,843 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.20.111.3
:65180 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheB
us], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.20.111.3:65180, coord_addr=
10.40.10.52:4775, is_server=true]]}
16:52:06,859 DEBUG PingWaiter:107 - waiting for initial members: time_to_wait=17
81, got 1 rsps
16:52:06,906 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.20.110.3:45282
16:52:06,906 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.20.111.3:6
5180, coord_addr=10.40.10.52:4775, is_server=true]
16:52:06,937 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.20.110.3
:45282 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheB
us], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.20.110.3:45282, coord_addr=
10.40.10.52:4775, is_server=true]]}
16:52:06,968 DEBUG PingWaiter:107 - waiting for initial members: time_to_wait=16
72, got 2 rsps
16:52:07,000 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.20.111.2:1620
16:52:07,000 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.20.110.3:4
5282, coord_addr=10.40.10.52:4775, is_server=true]
16:52:07,031 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.20.111.2
:1620 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.20.111.2:1620, coord_addr=10
.20.110.1:2715, is_server=true]]}
16:52:07,062 DEBUG PingWaiter:122 - initial mbrs are [[own_addr=10.20.111.1:1117
, coord_addr=10.20.111.1:1117, is_server=false], [own_addr=10.20.111.3:65180, co
ord_addr=10.40.10.52:4775, is_server=true], [own_addr=10.20.110.3:45282, coord_a
ddr=10.40.10.52:4775, is_server=true]]
16:52:07,109 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.10.52:4775
16:52:07,109 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.20.111.2:1
620, coord_addr=10.20.110.1:2715, is_server=true]
16:52:07,156 DEBUG ClientGmsImpl:66 - initial_mbrs are [[own_addr=10.20.111.1:11
17, coord_addr=10.20.111.1:1117, is_server=false], [own_addr=10.20.111.3:65180,
coord_addr=10.40.10.52:4775, is_server=true], [own_addr=10.20.110.3:45282, coord
_addr=10.40.10.52:4775, is_server=true]]
16:52:07,156 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.10.52
:4775 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.10.52:4775, coord_addr=10
.20.110.1:2715, is_server=true]]}
16:52:07,234 DEBUG ClientGmsImpl:113 - sending handleJoin(10.20.111.1:1172) to 1
0.40.10.52:4775
16:52:07,265 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.125.3:1739
16:52:07,265 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.10.52:4
775, coord_addr=10.20.110.1:2715, is_server=true]
16:52:07,296 DEBUG UNICAST:222 - [10.20.111.1:1172] --> DATA(10.40.10.52:4775: #
83, first=false)
16:52:07,312 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.125.3
:1739 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.125.3:1739, coord_addr=10
.20.110.1:2715, is_server=true]]}
16:52:07,359 DEBUG UDP:803 - sending msg to 10.40.10.52:4775 (src=10.20.111.1:11
72), headers are {UNICAST=[UNICAST: DATA, seqno=83], GMS=GmsHeader[JOIN_REQ]: mb
r=10.20.111.1:1172, UDP=[UDP:channel_name=OSCacheBus]}
16:52:07,406 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.121.3:4249
16:52:07,406 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.125.3:1
739, coord_addr=10.20.110.1:2715, is_server=true]
16:52:07,453 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.121.3
:4249 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.121.3:4249, coord_addr=10
.20.110.1:2715, is_server=true]]}
16:52:07,515 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.10.51:3426
16:52:07,515 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.121.3:4
249, coord_addr=10.20.110.1:2715, is_server=true]
16:52:07,531 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.10.51
:3426 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.10.51:3426, coord_addr=10
.20.110.1:2715, is_server=true]]}
16:52:07,593 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.10.51:3
426, coord_addr=10.20.110.1:2715, is_server=true]
16:52:07,593 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.126.2:4228
16:52:07,625 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.126.2
:4228 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.126.2:4228, coord_addr=10
.20.110.1:2715, is_server=true]]}
16:52:07,671 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.126.1:58260
16:52:07,671 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.126.2:4
228, coord_addr=10.20.110.1:2715, is_server=true]
16:52:07,687 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.126.1
:58260 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheB
us], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.126.1:58260, coord_addr=
10.20.110.1:2715, is_server=true]]}
16:52:07,750 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.125.4:2799
16:52:07,750 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.126.1:5
8260, coord_addr=10.20.110.1:2715, is_server=true]
16:52:07,765 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.125.4
:2799 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.125.4:2799, coord_addr=10
.20.110.1:2715, is_server=true]]}
16:52:07,828 DEBUG PING:203 - received GET_MBRS_RSP, rsp=[own_addr=10.40.125.4:2
799, coord_addr=10.20.110.1:2715, is_server=true]
16:52:07,828 DEBUG UDP:1542 - received (ucast) 78 bytes from /10.40.121.1:3810
16:52:07,859 DEBUG UDP:757 - message is [dst: 10.20.111.1:1172, src: 10.40.121.1
:3810 (2 headers), size = 0 bytes], headers are {UDP=[UDP:channel_name=OSCacheBu
s], PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=10.40.121.1:3810, coord_addr=10
.20.110.1:2715, is_server=true]]}
Terminate batch job (Y/N)? y

D:\sbroot\r6.1.0.0\Bin>

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

[JBoss JIRA] Created: (JGRP-343) Unable to bring additional node to a cluster due to WARN ClientGmsImpl:331 - there was more than 1 candidate for coord and WARN ClientGmsImpl:118 - join(10.20.111.1:1362) failed, retrying