[infinispan-issues] [JBoss JIRA] (ISPN-7491) When a node leaves the cluster it emits unsolicited state

When a node leaves the cluster it emits unsolicited state --------------------------------------------------------- Key: ISPN-7491 URL: https://issues.jboss.org/browse/ISPN-7491 Project: Infinispan Issue Type: Bug Components: Cloud Integrations, Core Affects Versions: 9.0.0.CR1 Environment: * OpenShift {{v1.5.0-alpha.2+e4b43ee}} * Custom Infinispan Server build (based on [these instructions|https://github.com/slaskawi/infinispan-1/tree/custom_image]). SHA1 {{2b0731b21649a88a75ed71d21b9cc06ba365e947}} Reporter: Sebastian Łaskawiec Priority: Blocker When I was performing [Spring Session and Kubernetes Rolling Update demo|https://bluejeans.com/s/pYKUg/] I encountered a couple of problems. One of the is this: {noformat} [transactions-repository-2-1gk60] 18:26:44,264 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t22) ISPN000336: Finished cluster-wide rebalance for cache ___protobuf_metadata, topology id = 18 [transactions-repository-2-pd4c7] [GC (Allocation Failure) 756168K->489129K(1013632K), 0.0439983 secs] [transactions-repository-2-1gk60] [GC (Allocation Failure) 686020K->416468K(1013632K), 0.0514641 secs] [transactions-repository-2-pd4c7] [GC (Allocation Failure) 768745K->501140K(1013632K), 0.0544712 secs] [transactions-repository-2-1gk60] [GC (Allocation Failure) 696084K->427206K(1013632K), 0.0997416 secs] [transactions-repository-2-pd4c7] [GC (Allocation Failure) 780756K->516161K(1013632K), 0.0674455 secs] [transactions-repository-2-jms3k] *** JBossAS process (81) received TERM signal *** [transactions-repository-2-jms3k] 18:26:46,371 INFO [org.jboss.as.server] (Thread-2) WFLYSRV0220: Server shutdown has been requested via an OS signal [transactions-repository-2-jms3k] 18:26:46,392 INFO [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-5) WFLYJCA0010: Unbound data source [java:jboss/datasources/ExampleDS] [transactions-repository-2-1gk60] 18:26:46,395 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t25) ISPN000310: Starting cluster-wide rebalance for cache sessions, topology CacheTopology{id=25, rebalanceId=9, currentCH=DefaultConsistentHash{ns=20, owners = (3)[transactions-repository-2-1gk60: 5+6, transactions-repository-2-pd4c7: 7+3, transactions-repository-3-6mzvd: 8+3]}, pendingCH=DefaultConsistentHash{ns=20, owners = (3)[transactions-repository-2-1gk60: 7+8, transactions-repository-2-pd4c7: 7+6, transactions-repository-3-6mzvd: 6+6]}, unionCH=null, actualMembers=[transactions-repository-2-1gk60, transactions-repository-2-pd4c7, transactions-repository-3-6mzvd], persistentUUIDs=[d47f6f69-97e8-4385-abd0-f1d43312d8c4, a17aa2e7-5137-4b5b-bf90-266207db7bb2, a000aa28-b1bd-453b-86e9-1628da898c36]} [transactions-repository-2-1gk60] 18:26:46,395 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t25) [Context=sessions][Context=transactions-repository-2-1gk60]ISPN100002: Started local rebalance [transactions-repository-2-pd4c7] 18:26:46,402 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t53) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions [transactions-repository-2-pd4c7] 18:26:46,411 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t24) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions [transactions-repository-2-pd4c7] 18:26:46,417 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t53) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions [transactions-repository-2-jms3k] 18:26:46,430 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-6) DGISPN0002: Stopped sessions cache from clustered container [transactions-repository-2-pd4c7] 18:26:46,441 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t24) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions [transactions-repository-2-pd4c7] 18:26:46,444 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t24) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions [transactions-repository-2-pd4c7] 18:26:46,444 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t54) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions [transactions-repository-2-pd4c7] 18:26:46,448 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t54) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions [transactions-repository-2-pd4c7] 18:26:46,450 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t54) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions [transactions-repository-2-pd4c7] 18:26:46,450 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t55) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions [transactions-repository-2-pd4c7] 18:26:46,451 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t54) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions [transactions-repository-2-pd4c7] 18:26:46,455 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t54) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions [transactions-repository-2-pd4c7] 18:26:46,456 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t24) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions [transactions-repository-2-pd4c7] 18:26:46,453 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t55) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions [transactions-repository-2-pd4c7] 18:26:46,466 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t55) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions [transactions-repository-2-pd4c7] 18:26:46,470 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t55) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions [transactions-repository-2-pd4c7] 18:26:46,471 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t58) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions [transactions-repository-2-pd4c7] 18:26:46,466 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t54) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions [transactions-repository-2-pd4c7] 18:26:46,473 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t56) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions [transactions-repository-2-pd4c7] 18:26:46,495 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions [transactions-repository-2-pd4c7] 18:26:46,497 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t63) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions [transactions-repository-2-pd4c7] 18:26:46,473 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t60) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions [transactions-repository-2-pd4c7] 18:26:46,494 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t61) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions [transactions-repository-2-pd4c7] 18:26:46,490 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t62) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions [transactions-repository-2-pd4c7] 18:26:46,477 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t64) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions [transactions-repository-2-pd4c7] 18:26:46,476 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t24) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions [transactions-repository-2-pd4c7] 18:26:46,476 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t55) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions [transactions-repository-2-pd4c7] 18:26:46,475 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t57) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions [transactions-repository-2-pd4c7] 18:26:46,475 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t59) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions [transactions-repository-2-pd4c7] 18:26:46,475 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t54) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions [transactions-repository-2-pd4c7] 18:26:46,474 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t58) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions [transactions-repository-2-1gk60] 18:26:46,513 INFO [org.infinispan.CLUSTER] (transport-thread--p4-t6) [Context=sessions][Context=transactions-repository-2-1gk60]ISPN100003: Finished local rebalance [transactions-repository-2-pd4c7] 18:26:46,514 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t57) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions [transactions-repository-2-pd4c7] 18:26:46,514 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t60) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions [transactions-repository-2-pd4c7] 18:26:46,515 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t60) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions [transactions-repository-2-pd4c7] 18:26:46,515 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t55) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions [transactions-repository-2-pd4c7] 18:26:46,515 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions [transactions-repository-2-pd4c7] 18:26:46,515 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t58) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions [transactions-repository-2-pd4c7] 18:26:46,516 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t56) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions [transactions-repository-2-pd4c7] 18:26:46,516 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t62) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions [transactions-repository-2-pd4c7] 18:26:46,516 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t55) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions [transactions-repository-2-1gk60] 18:26:46,527 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t14) [Context=sessions][Context=transactions-repository-2-pd4c7]ISPN100003: Finished local rebalance [transactions-repository-2-pd4c7] 18:26:46,517 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t64) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions [transactions-repository-2-pd4c7] 18:26:46,517 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions [transactions-repository-2-jms3k] 18:26:46,529 INFO [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-8) WFLYJCA0019: Stopped Driver service with driver-name = h2 [transactions-repository-2-pd4c7] 18:26:46,536 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions [transactions-repository-2-pd4c7] 18:26:46,541 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t65) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions [transactions-repository-2-1gk60] 18:26:46,548 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t19) [Context=sessions][Context=transactions-repository-3-6mzvd]ISPN100003: Finished local rebalance [transactions-repository-2-1gk60] 18:26:46,549 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t19) ISPN000336: Finished cluster-wide rebalance for cache sessions, topology id = 25 [transactions-repository-2-pd4c7] 18:26:46,558 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t53) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions [transactions-repository-2-pd4c7] 18:26:46,558 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions [transactions-repository-2-1gk60] 18:26:46,574 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t10) ISPN000310: Starting cluster-wide rebalance for cache default, topology CacheTopology{id=21, rebalanceId=10, currentCH=DefaultConsistentHash{ns=20, owners = (3)[transactions-repository-2-1gk60: 5+6, transactions-repository-2-pd4c7: 7+3, transactions-repository-3-6mzvd: 8+3]}, pendingCH=DefaultConsistentHash{ns=20, owners = (3)[transactions-repository-2-1gk60: 7+8, transactions-repository-2-pd4c7: 7+6, transactions-repository-3-6mzvd: 6+6]}, unionCH=null, actualMembers=[transactions-repository-2-1gk60, transactions-repository-2-pd4c7, transactions-repository-3-6mzvd], persistentUUIDs=[d47f6f69-97e8-4385-abd0-f1d43312d8c4, a17aa2e7-5137-4b5b-bf90-266207db7bb2, a000aa28-b1bd-453b-86e9-1628da898c36]} [transactions-repository-2-1gk60] 18:26:46,574 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t10) [Context=default][Context=transactions-repository-2-1gk60]ISPN100002: Started local rebalance [transactions-repository-2-pd4c7] 18:26:46,536 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t64) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions [transactions-repository-2-pd4c7] 18:26:46,563 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions [transactions-repository-2-pd4c7] 18:26:46,517 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t58) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions [transactions-repository-2-pd4c7] 18:26:46,563 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t53) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions [transactions-repository-2-pd4c7] 18:26:46,517 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t63) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions [transactions-repository-2-1gk60] 18:26:46,591 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t2) ISPN000310: Starting cluster-wide rebalance for cache memcachedCache, topology CacheTopology{id=21, rebalanceId=10, currentCH=DefaultConsistentHash{ns=20, owners = (3)[transactions-repository-2-1gk60: 5+6, transactions-repository-2-pd4c7: 7+3, transactions-repository-3-6mzvd: 8+3]}, pendingCH=DefaultConsistentHash{ns=20, owners = (3)[transactions-repository-2-1gk60: 7+8, transactions-repository-2-pd4c7: 7+6, transactions-repository-3-6mzvd: 6+6]}, unionCH=null, actualMembers=[transactions-repository-2-1gk60, transactions-repository-2-pd4c7, transactions-repository-3-6mzvd], persistentUUIDs=[d47f6f69-97e8-4385-abd0-f1d43312d8c4, a17aa2e7-5137-4b5b-bf90-266207db7bb2, a000aa28-b1bd-453b-86e9-1628da898c36]} [transactions-repository-2-1gk60] 18:26:46,596 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t2) [Context=memcachedCache][Context=transactions-repository-2-1gk60]ISPN100002: Started local rebalance [transactions-repository-2-1gk60] 18:26:46,602 INFO [org.infinispan.CLUSTER] (transport-thread--p4-t25) [Context=default][Context=transactions-repository-2-1gk60]ISPN100003: Finished local rebalance [transactions-repository-2-jms3k] 18:26:46,599 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-7) DGISPN0002: Stopped default cache from clustered container [transactions-repository-2-pd4c7] 18:26:46,609 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t58) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions [transactions-repository-2-pd4c7] 18:26:46,610 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions [transactions-repository-2-pd4c7] 18:26:46,610 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t63) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions [transactions-repository-2-pd4c7] 18:26:46,610 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t64) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions [transactions-repository-2-pd4c7] 18:26:46,611 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions [transactions-repository-2-pd4c7] 18:26:46,611 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t64) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions [transactions-repository-2-pd4c7] 18:26:46,611 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions [transactions-repository-2-pd4c7] 18:26:46,611 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t58) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions [transactions-repository-2-pd4c7] 18:26:46,612 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t55) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions [transactions-repository-2-pd4c7] 18:26:46,611 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t64) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions [transactions-repository-2-pd4c7] 18:26:46,611 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t63) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions [transactions-repository-2-pd4c7] 18:26:46,612 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions [transactions-repository-2-1gk60] 18:26:46,635 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t6) [Context=default][Context=transactions-repository-2-pd4c7]ISPN100003: Finished local rebalance [transactions-repository-2-1gk60] 18:26:46,640 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t15) [Context=default][Context=transactions-repository-3-6mzvd]ISPN100003: Finished local rebalance [transactions-repository-2-1gk60] 18:26:46,642 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t15) ISPN000336: Finished cluster-wide rebalance for cache default, topology id = 21 [transactions-repository-2-1gk60] 18:26:46,644 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t20) [Context=memcachedCache][Context=transactions-repository-2-pd4c7]ISPN100003: Finished local rebalance [transactions-repository-2-pd4c7] 18:26:46,641 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t64) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions [transactions-repository-2-pd4c7] 18:26:46,641 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t55) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions [transactions-repository-2-pd4c7] 18:26:46,641 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions [transactions-repository-2-pd4c7] 18:26:46,641 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t53) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions [transactions-repository-2-pd4c7] 18:26:46,641 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t63) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions [transactions-repository-2-pd4c7] 18:26:46,641 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t58) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions [transactions-repository-2-pd4c7] 18:26:46,641 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions [transactions-repository-2-pd4c7] 18:26:46,641 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t65) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions [transactions-repository-2-pd4c7] 18:26:46,641 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t64) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions [transactions-repository-2-pd4c7] 18:26:46,641 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t62) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions [transactions-repository-2-1gk60] 18:26:46,661 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t9) [Context=memcachedCache][Context=transactions-repository-3-6mzvd]ISPN100003: Finished local rebalance [transactions-repository-2-1gk60] 18:26:46,663 INFO [org.infinispan.CLUSTER] (transport-thread--p4-t22) [Context=memcachedCache][Context=transactions-repository-2-1gk60]ISPN100003: Finished local rebalance [transactions-repository-2-1gk60] 18:26:46,664 INFO [org.infinispan.CLUSTER] (transport-thread--p4-t22) ISPN000336: Finished cluster-wide rebalance for cache memcachedCache, topology id = 21 [transactions-repository-2-pd4c7] 18:26:46,668 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t56) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions [transactions-repository-2-pd4c7] 18:26:46,668 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t65) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions [transactions-repository-2-jms3k] 18:26:46,673 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-4) DGISPN0002: Stopped memcachedCache cache from clustered container [transactions-repository-2-pd4c7] 18:26:46,674 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t56) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions [transactions-repository-2-pd4c7] 18:26:46,675 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t65) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions [transactions-repository-2-pd4c7] 18:26:46,679 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t56) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions [transactions-repository-2-pd4c7] 18:26:46,679 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t65) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions [transactions-repository-2-pd4c7] 18:26:46,686 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t65) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions [transactions-repository-2-pd4c7] 18:26:46,687 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t56) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions [transactions-repository-2-pd4c7] 18:26:46,713 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t56) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions [transactions-repository-2-pd4c7] 18:26:46,715 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t65) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions [transactions-repository-2-pd4c7] 18:26:46,722 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions [transactions-repository-2-pd4c7] 18:26:46,722 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t56) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions [transactions-repository-2-pd4c7] 18:26:46,728 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions [transactions-repository-2-pd4c7] 18:26:46,728 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t56) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions [transactions-repository-2-pd4c7] 18:26:46,729 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t56) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions [transactions-repository-2-pd4c7] 18:26:46,729 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions [transactions-repository-2-pd4c7] 18:26:46,747 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t56) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions [transactions-repository-2-pd4c7] 18:26:46,748 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions [transactions-repository-2-pd4c7] 18:26:46,753 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t56) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions [transactions-repository-2-pd4c7] 18:26:46,753 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions [transactions-repository-2-pd4c7] 18:26:46,755 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t56) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions [transactions-repository-2-pd4c7] 18:26:46,756 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions [transactions-repository-2-pd4c7] 18:26:46,757 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t56) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions [transactions-repository-2-pd4c7] 18:26:46,758 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions [transactions-repository-2-pd4c7] 18:26:46,759 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t56) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions [transactions-repository-2-pd4c7] 18:26:46,760 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions [transactions-repository-2-pd4c7] 18:26:46,760 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions [transactions-repository-2-pd4c7] 18:26:46,760 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t56) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions [transactions-repository-2-pd4c7] 18:26:46,763 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t64) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions [transactions-repository-2-pd4c7] 18:26:46,765 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t62) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions [transactions-repository-2-pd4c7] 18:26:46,765 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t65) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions [transactions-repository-2-pd4c7] 18:26:46,766 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t58) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions [transactions-repository-2-pd4c7] 18:26:46,766 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t63) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions [transactions-repository-2-pd4c7] 18:26:46,765 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t53) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions [transactions-repository-2-pd4c7] 18:26:46,763 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t56) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions [transactions-repository-2-pd4c7] 18:26:46,764 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions [transactions-repository-2-pd4c7] 18:26:46,774 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t53) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions [transactions-repository-2-pd4c7] 18:26:46,774 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t56) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions [transactions-repository-2-pd4c7] 18:26:46,777 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t56) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions [transactions-repository-2-pd4c7] 18:26:46,777 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t53) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions [transactions-repository-2-pd4c7] 18:26:46,778 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t53) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions [transactions-repository-2-pd4c7] 18:26:46,778 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions [transactions-repository-2-pd4c7] 18:26:46,778 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t63) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions [transactions-repository-2-jms3k] 18:26:46,778 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-2) DGISPN0002: Stopped transactions cache from clustered container [transactions-repository-2-pd4c7] 18:26:46,778 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t56) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions [transactions-repository-2-pd4c7] 18:26:46,784 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t63) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions [transactions-repository-2-pd4c7] 18:26:46,784 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t56) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions [transactions-repository-2-pd4c7] 18:26:46,787 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t63) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions [transactions-repository-2-pd4c7] 18:26:46,787 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t56) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions [transactions-repository-2-1gk60] 18:26:46,802 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 6 of cache transactions {noformat} Full logs from Rolling Update process might be found here: https://gist.github.com/slaskawi/d789431ab5f16d46136dafffd7d4a89e Steps to reproduce: * Start local OpenShift Cluster * invoke `./init_infrastructure.sh` from https://github.com/slaskawi/presentations/tree/ISPN-7487-reproducer * invoke `cd transaction-creator && mvn fabric8:run` * Start Spring Session Demo `cd session-demo && mvn fabric8:run` * Create a client which inserts data (`watch -n 0.5 curl http://<spring-session-demo-pod-ip>/sessions`) and at the same time invoke the rolling update: `oc deploy transactions-repository --latest -n myproject` * Observe logs `kubetail -l environment=infrastructure`