UFC and MFC headers get mixed up -------------------------------- Key: JGRP-2072 URL: https://issues.jboss.org/browse/JGRP-2072 Project: JGroups Issue Type: Bug Affects Versions: 3.6.9 Reporter: Dan Berindei Assignee: Bela Ban Fix For: 3.6.10, 4.0 {noformat} 11:24:19,178 TRACE (OOB-10,NodeD-30623:[]) [MFC] NodeA-6644 used 4969 credits, 396763 remaining 11:24:19,238 TRACE (OOB-10,NodeD-30623:[]) [MFC] sending 603237 credits to NodeA-6644 11:24:19,238 TRACE (OOB-10,NodeD-30623:[]) [UDP] NodeD-30623: sending msg to NodeA-6644, src=NodeD-30623, headers are MFC: REPLENISH, UNICAST3: DATA, seqno=8825, TP: [cluster_name=default] 11:24:19,246 TRACE (INT-1,NodeA-6644:[]) [UDP] NodeA-6644: received [dst: NodeA-6644, src: NodeD-30623 (3 headers), size=8 bytes, flags=OOB|DONT_BUNDLE|INTERNAL], headers are UFC: REPLENISH, UNICAST3: DATA, seqno=8825, TP: [cluster_name=default] 11:24:19,246 TRACE (INT-1,NodeA-6644:[]) [UFC] received 603237 credits from NodeD-30623, old credits: 938772, new credits: 1000000 {noformat} {noformat} 15:20:14,596 TRACE (OOB-44,NodeC-64981:[]) [UFC] NodeA-12977 used 2305 credits, 798175 remaining 15:20:14,614 TRACE (OOB-44,NodeC-64981:[]) [UFC] sending 1201825 credits to NodeA-12977 15:20:14,614 TRACE (OOB-44,NodeC-64981:[]) [UDP] NodeC-64981: sending msg to NodeA-12977, src=NodeC-64981, headers are UFC: REPLENISH, UNICAST3: DATA, seqno=34854, TP: [cluster_name=default] 15:20:14,629 TRACE (INT-1,NodeA-12977:[]) [UDP] NodeA-12977: received [dst: NodeA-12977, src: NodeC-64981 (3 headers), size=8 bytes, flags=OOB|DONT_BUNDLE|INTERNAL], headers are MFC: REPLENISH, UNICAST3: DATA, seqno=34854, TP: [cluster_name=default] 15:20:14,629 TRACE (INT-1,NodeA-12977:[]) [UFC] NodeC-64981 used 8 credits, 1859662 remaining 15:20:14,629 TRACE (INT-1,NodeA-12977:[]) [MFC] received 1201825 credits from NodeC-64981, new credits for NodeC-64981: 2000000, min_credits=793978 {noformat} UFC/MFC expect their messages to be reliable, so they don't repeat {{REPLENISH}} messages, and the only way to replenish the credits is to send a {{CREDIT_REQUEST}} message. But UFC/MFC only send a credit request after all the threads trying to send a request or response to the affected destination block for {{max_block_time}}. I was able to reproduce the problem with an Infinispan [micro-benchmark|https://github.com/danberindei/infinispan-microbenchmarks]: a cluster of 4 nodes, a replicated cache, 400 threads writing to the cache. It happens both with JGroups 3.6.9.Final and the latest master.