[JBoss JIRA] (ISPN-7491) When a node leaves the cluster it emits unsolicited state
by Sebastian Łaskawiec (JIRA)
[ https://issues.jboss.org/browse/ISPN-7491?page=com.atlassian.jira.plugin.... ]
Sebastian Łaskawiec reassigned ISPN-7491:
-----------------------------------------
Assignee: Sebastian Łaskawiec
> When a node leaves the cluster it emits unsolicited state
> ---------------------------------------------------------
>
> Key: ISPN-7491
> URL: https://issues.jboss.org/browse/ISPN-7491
> Project: Infinispan
> Issue Type: Bug
> Components: Cloud Integrations, Core
> Affects Versions: 9.0.0.CR1
> Environment: * OpenShift {{v1.5.0-alpha.2+e4b43ee}}
> * Custom Infinispan Server build (based on [these instructions|https://github.com/slaskawi/infinispan-1/tree/custom_image]). SHA1 {{2b0731b21649a88a75ed71d21b9cc06ba365e947}}
> Reporter: Sebastian Łaskawiec
> Assignee: Sebastian Łaskawiec
> Priority: Blocker
>
> When I was performing [Spring Session and Kubernetes Rolling Update demo|https://bluejeans.com/s/pYKUg/] I encountered a couple of problems.
> One of the is this:
> {noformat}
> [transactions-repository-2-1gk60] 18:26:44,264 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t22) ISPN000336: Finished cluster-wide rebalance for cache ___protobuf_metadata, topology id = 18
> [transactions-repository-2-pd4c7] [GC (Allocation Failure) 756168K->489129K(1013632K), 0.0439983 secs]
> [transactions-repository-2-1gk60] [GC (Allocation Failure) 686020K->416468K(1013632K), 0.0514641 secs]
> [transactions-repository-2-pd4c7] [GC (Allocation Failure) 768745K->501140K(1013632K), 0.0544712 secs]
> [transactions-repository-2-1gk60] [GC (Allocation Failure) 696084K->427206K(1013632K), 0.0997416 secs]
> [transactions-repository-2-pd4c7] [GC (Allocation Failure) 780756K->516161K(1013632K), 0.0674455 secs]
> [transactions-repository-2-jms3k] *** JBossAS process (81) received TERM signal ***
> [transactions-repository-2-jms3k] 18:26:46,371 INFO [org.jboss.as.server] (Thread-2) WFLYSRV0220: Server shutdown has been requested via an OS signal
> [transactions-repository-2-jms3k] 18:26:46,392 INFO [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-5) WFLYJCA0010: Unbound data source [java:jboss/datasources/ExampleDS]
> [transactions-repository-2-1gk60] 18:26:46,395 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t25) ISPN000310: Starting cluster-wide rebalance for cache sessions, topology CacheTopology{id=25, rebalanceId=9, currentCH=DefaultConsistentHash{ns=20, owners = (3)[transactions-repository-2-1gk60: 5+6, transactions-repository-2-pd4c7: 7+3, transactions-repository-3-6mzvd: 8+3]}, pendingCH=DefaultConsistentHash{ns=20, owners = (3)[transactions-repository-2-1gk60: 7+8, transactions-repository-2-pd4c7: 7+6, transactions-repository-3-6mzvd: 6+6]}, unionCH=null, actualMembers=[transactions-repository-2-1gk60, transactions-repository-2-pd4c7, transactions-repository-3-6mzvd], persistentUUIDs=[d47f6f69-97e8-4385-abd0-f1d43312d8c4, a17aa2e7-5137-4b5b-bf90-266207db7bb2, a000aa28-b1bd-453b-86e9-1628da898c36]}
> [transactions-repository-2-1gk60] 18:26:46,395 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t25) [Context=sessions][Context=transactions-repository-2-1gk60]ISPN100002: Started local rebalance
> [transactions-repository-2-pd4c7] 18:26:46,402 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t53) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,411 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t24) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,417 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t53) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-jms3k] 18:26:46,430 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-6) DGISPN0002: Stopped sessions cache from clustered container
> [transactions-repository-2-pd4c7] 18:26:46,441 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t24) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,444 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t24) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,444 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t54) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,448 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t54) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,450 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t54) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,450 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t55) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,451 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t54) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,455 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t54) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,456 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t24) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,453 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t55) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,466 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t55) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,470 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t55) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,471 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t58) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,466 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t54) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,473 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t56) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,495 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,497 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t63) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,473 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t60) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,494 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t61) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,490 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t62) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,477 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t64) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,476 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t24) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,476 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t55) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,475 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t57) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,475 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t59) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,475 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t54) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,474 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t58) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-1gk60] 18:26:46,513 INFO [org.infinispan.CLUSTER] (transport-thread--p4-t6) [Context=sessions][Context=transactions-repository-2-1gk60]ISPN100003: Finished local rebalance
> [transactions-repository-2-pd4c7] 18:26:46,514 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t57) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,514 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t60) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,515 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t60) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,515 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t55) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,515 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,515 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t58) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,516 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t56) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,516 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t62) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,516 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t55) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-1gk60] 18:26:46,527 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t14) [Context=sessions][Context=transactions-repository-2-pd4c7]ISPN100003: Finished local rebalance
> [transactions-repository-2-pd4c7] 18:26:46,517 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t64) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,517 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-jms3k] 18:26:46,529 INFO [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-8) WFLYJCA0019: Stopped Driver service with driver-name = h2
> [transactions-repository-2-pd4c7] 18:26:46,536 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,541 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t65) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-1gk60] 18:26:46,548 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t19) [Context=sessions][Context=transactions-repository-3-6mzvd]ISPN100003: Finished local rebalance
> [transactions-repository-2-1gk60] 18:26:46,549 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t19) ISPN000336: Finished cluster-wide rebalance for cache sessions, topology id = 25
> [transactions-repository-2-pd4c7] 18:26:46,558 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t53) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,558 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-1gk60] 18:26:46,574 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t10) ISPN000310: Starting cluster-wide rebalance for cache default, topology CacheTopology{id=21, rebalanceId=10, currentCH=DefaultConsistentHash{ns=20, owners = (3)[transactions-repository-2-1gk60: 5+6, transactions-repository-2-pd4c7: 7+3, transactions-repository-3-6mzvd: 8+3]}, pendingCH=DefaultConsistentHash{ns=20, owners = (3)[transactions-repository-2-1gk60: 7+8, transactions-repository-2-pd4c7: 7+6, transactions-repository-3-6mzvd: 6+6]}, unionCH=null, actualMembers=[transactions-repository-2-1gk60, transactions-repository-2-pd4c7, transactions-repository-3-6mzvd], persistentUUIDs=[d47f6f69-97e8-4385-abd0-f1d43312d8c4, a17aa2e7-5137-4b5b-bf90-266207db7bb2, a000aa28-b1bd-453b-86e9-1628da898c36]}
> [transactions-repository-2-1gk60] 18:26:46,574 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t10) [Context=default][Context=transactions-repository-2-1gk60]ISPN100002: Started local rebalance
> [transactions-repository-2-pd4c7] 18:26:46,536 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t64) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,563 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,517 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t58) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,563 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t53) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,517 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t63) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-1gk60] 18:26:46,591 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t2) ISPN000310: Starting cluster-wide rebalance for cache memcachedCache, topology CacheTopology{id=21, rebalanceId=10, currentCH=DefaultConsistentHash{ns=20, owners = (3)[transactions-repository-2-1gk60: 5+6, transactions-repository-2-pd4c7: 7+3, transactions-repository-3-6mzvd: 8+3]}, pendingCH=DefaultConsistentHash{ns=20, owners = (3)[transactions-repository-2-1gk60: 7+8, transactions-repository-2-pd4c7: 7+6, transactions-repository-3-6mzvd: 6+6]}, unionCH=null, actualMembers=[transactions-repository-2-1gk60, transactions-repository-2-pd4c7, transactions-repository-3-6mzvd], persistentUUIDs=[d47f6f69-97e8-4385-abd0-f1d43312d8c4, a17aa2e7-5137-4b5b-bf90-266207db7bb2, a000aa28-b1bd-453b-86e9-1628da898c36]}
> [transactions-repository-2-1gk60] 18:26:46,596 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t2) [Context=memcachedCache][Context=transactions-repository-2-1gk60]ISPN100002: Started local rebalance
> [transactions-repository-2-1gk60] 18:26:46,602 INFO [org.infinispan.CLUSTER] (transport-thread--p4-t25) [Context=default][Context=transactions-repository-2-1gk60]ISPN100003: Finished local rebalance
> [transactions-repository-2-jms3k] 18:26:46,599 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-7) DGISPN0002: Stopped default cache from clustered container
> [transactions-repository-2-pd4c7] 18:26:46,609 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t58) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,610 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,610 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t63) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,610 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t64) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,611 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,611 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t64) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,611 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,611 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t58) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,612 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t55) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,611 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t64) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,611 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t63) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,612 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-1gk60] 18:26:46,635 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t6) [Context=default][Context=transactions-repository-2-pd4c7]ISPN100003: Finished local rebalance
> [transactions-repository-2-1gk60] 18:26:46,640 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t15) [Context=default][Context=transactions-repository-3-6mzvd]ISPN100003: Finished local rebalance
> [transactions-repository-2-1gk60] 18:26:46,642 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t15) ISPN000336: Finished cluster-wide rebalance for cache default, topology id = 21
> [transactions-repository-2-1gk60] 18:26:46,644 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t20) [Context=memcachedCache][Context=transactions-repository-2-pd4c7]ISPN100003: Finished local rebalance
> [transactions-repository-2-pd4c7] 18:26:46,641 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t64) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,641 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t55) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,641 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,641 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t53) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,641 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t63) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,641 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t58) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,641 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,641 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t65) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,641 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t64) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,641 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t62) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-1gk60] 18:26:46,661 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t9) [Context=memcachedCache][Context=transactions-repository-3-6mzvd]ISPN100003: Finished local rebalance
> [transactions-repository-2-1gk60] 18:26:46,663 INFO [org.infinispan.CLUSTER] (transport-thread--p4-t22) [Context=memcachedCache][Context=transactions-repository-2-1gk60]ISPN100003: Finished local rebalance
> [transactions-repository-2-1gk60] 18:26:46,664 INFO [org.infinispan.CLUSTER] (transport-thread--p4-t22) ISPN000336: Finished cluster-wide rebalance for cache memcachedCache, topology id = 21
> [transactions-repository-2-pd4c7] 18:26:46,668 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t56) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,668 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t65) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-jms3k] 18:26:46,673 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-4) DGISPN0002: Stopped memcachedCache cache from clustered container
> [transactions-repository-2-pd4c7] 18:26:46,674 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t56) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,675 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t65) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,679 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t56) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,679 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t65) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,686 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t65) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,687 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t56) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,713 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t56) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,715 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t65) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,722 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,722 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t56) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,728 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,728 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t56) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,729 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t56) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,729 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,747 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t56) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,748 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,753 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t56) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,753 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,755 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t56) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,756 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,757 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t56) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,758 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,759 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t56) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,760 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,760 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,760 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t56) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,763 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t64) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,765 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t62) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,765 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t65) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,766 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t58) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,766 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t63) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,765 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t53) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,763 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t56) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,764 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,774 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t53) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,774 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t56) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,777 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t56) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,777 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t53) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,778 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t53) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,778 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,778 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t63) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-jms3k] 18:26:46,778 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-2) DGISPN0002: Stopped transactions cache from clustered container
> [transactions-repository-2-pd4c7] 18:26:46,778 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t56) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,784 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t63) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,784 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t56) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,787 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t63) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,787 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t56) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-1gk60] 18:26:46,802 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 6 of cache transactions
> {noformat}
> Full logs from Rolling Update process might be found here: https://gist.github.com/slaskawi/d789431ab5f16d46136dafffd7d4a89e
> Steps to reproduce:
> * Start local OpenShift Cluster
> * invoke `./init_infrastructure.sh` from https://github.com/slaskawi/presentations/tree/ISPN-7487-reproducer
> * invoke `cd transaction-creator && mvn fabric8:run`
> * Start Spring Session Demo `cd session-demo && mvn fabric8:run`
> * Create a client which inserts data (`watch -n 0.5 curl http://<spring-session-demo-pod-ip>/sessions`) and at the same time invoke the rolling update: `oc deploy transactions-repository --latest -n myproject`
> * Observe logs `kubetail -l environment=infrastructure`
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
9 years, 1 month
- 1
- 0
[JBoss JIRA] (ISPN-7491) When a node leaves the cluster it emits unsolicited state
by Sebastian Łaskawiec (JIRA)
[ https://issues.jboss.org/browse/ISPN-7491?page=com.atlassian.jira.plugin.... ]
Sebastian Łaskawiec updated ISPN-7491:
--------------------------------------
Status: Pull Request Sent (was: Open)
Git Pull Request: https://github.com/infinispan/infinispan/pull/4875
> When a node leaves the cluster it emits unsolicited state
> ---------------------------------------------------------
>
> Key: ISPN-7491
> URL: https://issues.jboss.org/browse/ISPN-7491
> Project: Infinispan
> Issue Type: Bug
> Components: Cloud Integrations, Core
> Affects Versions: 9.0.0.CR1
> Environment: * OpenShift {{v1.5.0-alpha.2+e4b43ee}}
> * Custom Infinispan Server build (based on [these instructions|https://github.com/slaskawi/infinispan-1/tree/custom_image]). SHA1 {{2b0731b21649a88a75ed71d21b9cc06ba365e947}}
> Reporter: Sebastian Łaskawiec
> Assignee: Sebastian Łaskawiec
> Priority: Blocker
>
> When I was performing [Spring Session and Kubernetes Rolling Update demo|https://bluejeans.com/s/pYKUg/] I encountered a couple of problems.
> One of the is this:
> {noformat}
> [transactions-repository-2-1gk60] 18:26:44,264 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t22) ISPN000336: Finished cluster-wide rebalance for cache ___protobuf_metadata, topology id = 18
> [transactions-repository-2-pd4c7] [GC (Allocation Failure) 756168K->489129K(1013632K), 0.0439983 secs]
> [transactions-repository-2-1gk60] [GC (Allocation Failure) 686020K->416468K(1013632K), 0.0514641 secs]
> [transactions-repository-2-pd4c7] [GC (Allocation Failure) 768745K->501140K(1013632K), 0.0544712 secs]
> [transactions-repository-2-1gk60] [GC (Allocation Failure) 696084K->427206K(1013632K), 0.0997416 secs]
> [transactions-repository-2-pd4c7] [GC (Allocation Failure) 780756K->516161K(1013632K), 0.0674455 secs]
> [transactions-repository-2-jms3k] *** JBossAS process (81) received TERM signal ***
> [transactions-repository-2-jms3k] 18:26:46,371 INFO [org.jboss.as.server] (Thread-2) WFLYSRV0220: Server shutdown has been requested via an OS signal
> [transactions-repository-2-jms3k] 18:26:46,392 INFO [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-5) WFLYJCA0010: Unbound data source [java:jboss/datasources/ExampleDS]
> [transactions-repository-2-1gk60] 18:26:46,395 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t25) ISPN000310: Starting cluster-wide rebalance for cache sessions, topology CacheTopology{id=25, rebalanceId=9, currentCH=DefaultConsistentHash{ns=20, owners = (3)[transactions-repository-2-1gk60: 5+6, transactions-repository-2-pd4c7: 7+3, transactions-repository-3-6mzvd: 8+3]}, pendingCH=DefaultConsistentHash{ns=20, owners = (3)[transactions-repository-2-1gk60: 7+8, transactions-repository-2-pd4c7: 7+6, transactions-repository-3-6mzvd: 6+6]}, unionCH=null, actualMembers=[transactions-repository-2-1gk60, transactions-repository-2-pd4c7, transactions-repository-3-6mzvd], persistentUUIDs=[d47f6f69-97e8-4385-abd0-f1d43312d8c4, a17aa2e7-5137-4b5b-bf90-266207db7bb2, a000aa28-b1bd-453b-86e9-1628da898c36]}
> [transactions-repository-2-1gk60] 18:26:46,395 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t25) [Context=sessions][Context=transactions-repository-2-1gk60]ISPN100002: Started local rebalance
> [transactions-repository-2-pd4c7] 18:26:46,402 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t53) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,411 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t24) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,417 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t53) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-jms3k] 18:26:46,430 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-6) DGISPN0002: Stopped sessions cache from clustered container
> [transactions-repository-2-pd4c7] 18:26:46,441 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t24) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,444 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t24) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,444 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t54) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,448 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t54) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,450 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t54) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,450 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t55) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,451 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t54) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,455 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t54) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,456 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t24) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,453 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t55) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,466 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t55) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,470 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t55) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,471 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t58) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,466 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t54) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,473 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t56) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,495 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,497 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t63) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,473 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t60) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,494 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t61) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,490 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t62) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,477 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t64) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,476 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t24) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,476 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t55) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,475 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t57) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,475 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t59) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,475 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t54) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,474 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t58) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-1gk60] 18:26:46,513 INFO [org.infinispan.CLUSTER] (transport-thread--p4-t6) [Context=sessions][Context=transactions-repository-2-1gk60]ISPN100003: Finished local rebalance
> [transactions-repository-2-pd4c7] 18:26:46,514 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t57) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,514 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t60) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,515 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t60) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,515 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t55) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,515 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,515 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t58) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,516 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t56) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,516 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t62) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,516 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t55) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-1gk60] 18:26:46,527 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t14) [Context=sessions][Context=transactions-repository-2-pd4c7]ISPN100003: Finished local rebalance
> [transactions-repository-2-pd4c7] 18:26:46,517 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t64) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,517 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-jms3k] 18:26:46,529 INFO [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-8) WFLYJCA0019: Stopped Driver service with driver-name = h2
> [transactions-repository-2-pd4c7] 18:26:46,536 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,541 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t65) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-1gk60] 18:26:46,548 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t19) [Context=sessions][Context=transactions-repository-3-6mzvd]ISPN100003: Finished local rebalance
> [transactions-repository-2-1gk60] 18:26:46,549 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t19) ISPN000336: Finished cluster-wide rebalance for cache sessions, topology id = 25
> [transactions-repository-2-pd4c7] 18:26:46,558 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t53) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,558 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-1gk60] 18:26:46,574 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t10) ISPN000310: Starting cluster-wide rebalance for cache default, topology CacheTopology{id=21, rebalanceId=10, currentCH=DefaultConsistentHash{ns=20, owners = (3)[transactions-repository-2-1gk60: 5+6, transactions-repository-2-pd4c7: 7+3, transactions-repository-3-6mzvd: 8+3]}, pendingCH=DefaultConsistentHash{ns=20, owners = (3)[transactions-repository-2-1gk60: 7+8, transactions-repository-2-pd4c7: 7+6, transactions-repository-3-6mzvd: 6+6]}, unionCH=null, actualMembers=[transactions-repository-2-1gk60, transactions-repository-2-pd4c7, transactions-repository-3-6mzvd], persistentUUIDs=[d47f6f69-97e8-4385-abd0-f1d43312d8c4, a17aa2e7-5137-4b5b-bf90-266207db7bb2, a000aa28-b1bd-453b-86e9-1628da898c36]}
> [transactions-repository-2-1gk60] 18:26:46,574 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t10) [Context=default][Context=transactions-repository-2-1gk60]ISPN100002: Started local rebalance
> [transactions-repository-2-pd4c7] 18:26:46,536 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t64) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,563 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,517 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t58) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,563 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t53) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,517 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t63) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-1gk60] 18:26:46,591 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t2) ISPN000310: Starting cluster-wide rebalance for cache memcachedCache, topology CacheTopology{id=21, rebalanceId=10, currentCH=DefaultConsistentHash{ns=20, owners = (3)[transactions-repository-2-1gk60: 5+6, transactions-repository-2-pd4c7: 7+3, transactions-repository-3-6mzvd: 8+3]}, pendingCH=DefaultConsistentHash{ns=20, owners = (3)[transactions-repository-2-1gk60: 7+8, transactions-repository-2-pd4c7: 7+6, transactions-repository-3-6mzvd: 6+6]}, unionCH=null, actualMembers=[transactions-repository-2-1gk60, transactions-repository-2-pd4c7, transactions-repository-3-6mzvd], persistentUUIDs=[d47f6f69-97e8-4385-abd0-f1d43312d8c4, a17aa2e7-5137-4b5b-bf90-266207db7bb2, a000aa28-b1bd-453b-86e9-1628da898c36]}
> [transactions-repository-2-1gk60] 18:26:46,596 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t2) [Context=memcachedCache][Context=transactions-repository-2-1gk60]ISPN100002: Started local rebalance
> [transactions-repository-2-1gk60] 18:26:46,602 INFO [org.infinispan.CLUSTER] (transport-thread--p4-t25) [Context=default][Context=transactions-repository-2-1gk60]ISPN100003: Finished local rebalance
> [transactions-repository-2-jms3k] 18:26:46,599 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-7) DGISPN0002: Stopped default cache from clustered container
> [transactions-repository-2-pd4c7] 18:26:46,609 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t58) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,610 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,610 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t63) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,610 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t64) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,611 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,611 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t64) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,611 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,611 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t58) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,612 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t55) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,611 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t64) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,611 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t63) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,612 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-1gk60] 18:26:46,635 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t6) [Context=default][Context=transactions-repository-2-pd4c7]ISPN100003: Finished local rebalance
> [transactions-repository-2-1gk60] 18:26:46,640 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t15) [Context=default][Context=transactions-repository-3-6mzvd]ISPN100003: Finished local rebalance
> [transactions-repository-2-1gk60] 18:26:46,642 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t15) ISPN000336: Finished cluster-wide rebalance for cache default, topology id = 21
> [transactions-repository-2-1gk60] 18:26:46,644 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t20) [Context=memcachedCache][Context=transactions-repository-2-pd4c7]ISPN100003: Finished local rebalance
> [transactions-repository-2-pd4c7] 18:26:46,641 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t64) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,641 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t55) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,641 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,641 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t53) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,641 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t63) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,641 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t58) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,641 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,641 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t65) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,641 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t64) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,641 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t62) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-1gk60] 18:26:46,661 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t9) [Context=memcachedCache][Context=transactions-repository-3-6mzvd]ISPN100003: Finished local rebalance
> [transactions-repository-2-1gk60] 18:26:46,663 INFO [org.infinispan.CLUSTER] (transport-thread--p4-t22) [Context=memcachedCache][Context=transactions-repository-2-1gk60]ISPN100003: Finished local rebalance
> [transactions-repository-2-1gk60] 18:26:46,664 INFO [org.infinispan.CLUSTER] (transport-thread--p4-t22) ISPN000336: Finished cluster-wide rebalance for cache memcachedCache, topology id = 21
> [transactions-repository-2-pd4c7] 18:26:46,668 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t56) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,668 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t65) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-jms3k] 18:26:46,673 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-4) DGISPN0002: Stopped memcachedCache cache from clustered container
> [transactions-repository-2-pd4c7] 18:26:46,674 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t56) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,675 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t65) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,679 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t56) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,679 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t65) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,686 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t65) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,687 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t56) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,713 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t56) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,715 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t65) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,722 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,722 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t56) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,728 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,728 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t56) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,729 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t56) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,729 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,747 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t56) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,748 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,753 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t56) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,753 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,755 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t56) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,756 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,757 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t56) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,758 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,759 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t56) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,760 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,760 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,760 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t56) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,763 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t64) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,765 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t62) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,765 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t65) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,766 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t58) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,766 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t63) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,765 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t53) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,763 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t56) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,764 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,774 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t53) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,774 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t56) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,777 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t56) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,777 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t53) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,778 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t53) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,778 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,778 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t63) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-jms3k] 18:26:46,778 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-2) DGISPN0002: Stopped transactions cache from clustered container
> [transactions-repository-2-pd4c7] 18:26:46,778 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t56) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,784 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t63) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,784 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t56) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,787 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t63) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,787 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t56) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-1gk60] 18:26:46,802 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 6 of cache transactions
> {noformat}
> Full logs from Rolling Update process might be found here: https://gist.github.com/slaskawi/d789431ab5f16d46136dafffd7d4a89e
> Steps to reproduce:
> * Start local OpenShift Cluster
> * invoke `./init_infrastructure.sh` from https://github.com/slaskawi/presentations/tree/ISPN-7487-reproducer
> * invoke `cd transaction-creator && mvn fabric8:run`
> * Start Spring Session Demo `cd session-demo && mvn fabric8:run`
> * Create a client which inserts data (`watch -n 0.5 curl http://<spring-session-demo-pod-ip>/sessions`) and at the same time invoke the rolling update: `oc deploy transactions-repository --latest -n myproject`
> * Observe logs `kubetail -l environment=infrastructure`
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
9 years, 1 month
- 1
- 0
[JBoss JIRA] (ISPN-7491) When a node leaves the cluster it emits unsolicited state
by Sebastian Łaskawiec (JIRA)
[ https://issues.jboss.org/browse/ISPN-7491?page=com.atlassian.jira.plugin.... ]
Sebastian Łaskawiec edited comment on ISPN-7491 at 2/22/17 12:38 AM:
---------------------------------------------------------------------
I've been talking about this with [~dan.berindei]. Here's the most important bit:
{quote}
Feb 20 15:54:24 <dberindei> slaskawi: it looks to me as if node transactions-repository-2-pd4c7 requested some state from transactions-repository-2-jms3k, but transactions-repository-2-jms3k left while it was sending state, so some state arrived on pd4c7 after the topology update from the coordinator that didn't include jms3k
Feb 20 15:54:31 <dberindei> slaskawi: so pretty harmless
{quote}
We decided to silence those errors out.
was (Author: sebastian.laskawiec):
I've been talking about this with [~dan.berindei]. Here's the most important bit:
{quote}
Feb 20 15:54:24 <dberindei> slaskawi: it looks to me as if node transactions-repository-2-pd4c7 requested some state from transactions-repository-2-jms3k, but transactions-repository-2-jms3k left while it was sending state, so some state arrived on pd4c7 after the topology update from the coordinator that didn't include jms3k
Feb 20 15:54:31 <dberindei> slaskawi: so pretty harmless
{quote}
> When a node leaves the cluster it emits unsolicited state
> ---------------------------------------------------------
>
> Key: ISPN-7491
> URL: https://issues.jboss.org/browse/ISPN-7491
> Project: Infinispan
> Issue Type: Bug
> Components: Cloud Integrations, Core
> Affects Versions: 9.0.0.CR1
> Environment: * OpenShift {{v1.5.0-alpha.2+e4b43ee}}
> * Custom Infinispan Server build (based on [these instructions|https://github.com/slaskawi/infinispan-1/tree/custom_image]). SHA1 {{2b0731b21649a88a75ed71d21b9cc06ba365e947}}
> Reporter: Sebastian Łaskawiec
> Priority: Blocker
>
> When I was performing [Spring Session and Kubernetes Rolling Update demo|https://bluejeans.com/s/pYKUg/] I encountered a couple of problems.
> One of the is this:
> {noformat}
> [transactions-repository-2-1gk60] 18:26:44,264 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t22) ISPN000336: Finished cluster-wide rebalance for cache ___protobuf_metadata, topology id = 18
> [transactions-repository-2-pd4c7] [GC (Allocation Failure) 756168K->489129K(1013632K), 0.0439983 secs]
> [transactions-repository-2-1gk60] [GC (Allocation Failure) 686020K->416468K(1013632K), 0.0514641 secs]
> [transactions-repository-2-pd4c7] [GC (Allocation Failure) 768745K->501140K(1013632K), 0.0544712 secs]
> [transactions-repository-2-1gk60] [GC (Allocation Failure) 696084K->427206K(1013632K), 0.0997416 secs]
> [transactions-repository-2-pd4c7] [GC (Allocation Failure) 780756K->516161K(1013632K), 0.0674455 secs]
> [transactions-repository-2-jms3k] *** JBossAS process (81) received TERM signal ***
> [transactions-repository-2-jms3k] 18:26:46,371 INFO [org.jboss.as.server] (Thread-2) WFLYSRV0220: Server shutdown has been requested via an OS signal
> [transactions-repository-2-jms3k] 18:26:46,392 INFO [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-5) WFLYJCA0010: Unbound data source [java:jboss/datasources/ExampleDS]
> [transactions-repository-2-1gk60] 18:26:46,395 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t25) ISPN000310: Starting cluster-wide rebalance for cache sessions, topology CacheTopology{id=25, rebalanceId=9, currentCH=DefaultConsistentHash{ns=20, owners = (3)[transactions-repository-2-1gk60: 5+6, transactions-repository-2-pd4c7: 7+3, transactions-repository-3-6mzvd: 8+3]}, pendingCH=DefaultConsistentHash{ns=20, owners = (3)[transactions-repository-2-1gk60: 7+8, transactions-repository-2-pd4c7: 7+6, transactions-repository-3-6mzvd: 6+6]}, unionCH=null, actualMembers=[transactions-repository-2-1gk60, transactions-repository-2-pd4c7, transactions-repository-3-6mzvd], persistentUUIDs=[d47f6f69-97e8-4385-abd0-f1d43312d8c4, a17aa2e7-5137-4b5b-bf90-266207db7bb2, a000aa28-b1bd-453b-86e9-1628da898c36]}
> [transactions-repository-2-1gk60] 18:26:46,395 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t25) [Context=sessions][Context=transactions-repository-2-1gk60]ISPN100002: Started local rebalance
> [transactions-repository-2-pd4c7] 18:26:46,402 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t53) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,411 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t24) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,417 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t53) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-jms3k] 18:26:46,430 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-6) DGISPN0002: Stopped sessions cache from clustered container
> [transactions-repository-2-pd4c7] 18:26:46,441 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t24) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,444 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t24) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,444 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t54) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,448 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t54) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,450 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t54) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,450 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t55) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,451 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t54) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,455 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t54) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,456 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t24) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,453 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t55) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,466 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t55) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,470 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t55) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,471 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t58) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,466 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t54) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,473 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t56) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,495 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,497 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t63) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,473 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t60) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,494 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t61) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,490 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t62) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,477 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t64) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,476 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t24) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,476 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t55) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,475 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t57) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,475 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t59) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,475 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t54) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,474 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t58) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-1gk60] 18:26:46,513 INFO [org.infinispan.CLUSTER] (transport-thread--p4-t6) [Context=sessions][Context=transactions-repository-2-1gk60]ISPN100003: Finished local rebalance
> [transactions-repository-2-pd4c7] 18:26:46,514 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t57) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,514 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t60) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,515 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t60) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,515 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t55) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,515 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,515 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t58) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,516 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t56) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,516 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t62) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,516 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t55) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-1gk60] 18:26:46,527 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t14) [Context=sessions][Context=transactions-repository-2-pd4c7]ISPN100003: Finished local rebalance
> [transactions-repository-2-pd4c7] 18:26:46,517 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t64) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,517 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-jms3k] 18:26:46,529 INFO [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-8) WFLYJCA0019: Stopped Driver service with driver-name = h2
> [transactions-repository-2-pd4c7] 18:26:46,536 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,541 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t65) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-1gk60] 18:26:46,548 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t19) [Context=sessions][Context=transactions-repository-3-6mzvd]ISPN100003: Finished local rebalance
> [transactions-repository-2-1gk60] 18:26:46,549 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t19) ISPN000336: Finished cluster-wide rebalance for cache sessions, topology id = 25
> [transactions-repository-2-pd4c7] 18:26:46,558 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t53) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,558 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-1gk60] 18:26:46,574 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t10) ISPN000310: Starting cluster-wide rebalance for cache default, topology CacheTopology{id=21, rebalanceId=10, currentCH=DefaultConsistentHash{ns=20, owners = (3)[transactions-repository-2-1gk60: 5+6, transactions-repository-2-pd4c7: 7+3, transactions-repository-3-6mzvd: 8+3]}, pendingCH=DefaultConsistentHash{ns=20, owners = (3)[transactions-repository-2-1gk60: 7+8, transactions-repository-2-pd4c7: 7+6, transactions-repository-3-6mzvd: 6+6]}, unionCH=null, actualMembers=[transactions-repository-2-1gk60, transactions-repository-2-pd4c7, transactions-repository-3-6mzvd], persistentUUIDs=[d47f6f69-97e8-4385-abd0-f1d43312d8c4, a17aa2e7-5137-4b5b-bf90-266207db7bb2, a000aa28-b1bd-453b-86e9-1628da898c36]}
> [transactions-repository-2-1gk60] 18:26:46,574 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t10) [Context=default][Context=transactions-repository-2-1gk60]ISPN100002: Started local rebalance
> [transactions-repository-2-pd4c7] 18:26:46,536 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t64) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,563 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,517 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t58) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,563 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t53) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,517 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t63) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-1gk60] 18:26:46,591 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t2) ISPN000310: Starting cluster-wide rebalance for cache memcachedCache, topology CacheTopology{id=21, rebalanceId=10, currentCH=DefaultConsistentHash{ns=20, owners = (3)[transactions-repository-2-1gk60: 5+6, transactions-repository-2-pd4c7: 7+3, transactions-repository-3-6mzvd: 8+3]}, pendingCH=DefaultConsistentHash{ns=20, owners = (3)[transactions-repository-2-1gk60: 7+8, transactions-repository-2-pd4c7: 7+6, transactions-repository-3-6mzvd: 6+6]}, unionCH=null, actualMembers=[transactions-repository-2-1gk60, transactions-repository-2-pd4c7, transactions-repository-3-6mzvd], persistentUUIDs=[d47f6f69-97e8-4385-abd0-f1d43312d8c4, a17aa2e7-5137-4b5b-bf90-266207db7bb2, a000aa28-b1bd-453b-86e9-1628da898c36]}
> [transactions-repository-2-1gk60] 18:26:46,596 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t2) [Context=memcachedCache][Context=transactions-repository-2-1gk60]ISPN100002: Started local rebalance
> [transactions-repository-2-1gk60] 18:26:46,602 INFO [org.infinispan.CLUSTER] (transport-thread--p4-t25) [Context=default][Context=transactions-repository-2-1gk60]ISPN100003: Finished local rebalance
> [transactions-repository-2-jms3k] 18:26:46,599 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-7) DGISPN0002: Stopped default cache from clustered container
> [transactions-repository-2-pd4c7] 18:26:46,609 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t58) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,610 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,610 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t63) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,610 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t64) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,611 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,611 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t64) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,611 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,611 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t58) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,612 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t55) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,611 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t64) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,611 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t63) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,612 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-1gk60] 18:26:46,635 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t6) [Context=default][Context=transactions-repository-2-pd4c7]ISPN100003: Finished local rebalance
> [transactions-repository-2-1gk60] 18:26:46,640 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t15) [Context=default][Context=transactions-repository-3-6mzvd]ISPN100003: Finished local rebalance
> [transactions-repository-2-1gk60] 18:26:46,642 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t15) ISPN000336: Finished cluster-wide rebalance for cache default, topology id = 21
> [transactions-repository-2-1gk60] 18:26:46,644 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t20) [Context=memcachedCache][Context=transactions-repository-2-pd4c7]ISPN100003: Finished local rebalance
> [transactions-repository-2-pd4c7] 18:26:46,641 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t64) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,641 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t55) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,641 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,641 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t53) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,641 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t63) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,641 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t58) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,641 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,641 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t65) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,641 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t64) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,641 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t62) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-1gk60] 18:26:46,661 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t9) [Context=memcachedCache][Context=transactions-repository-3-6mzvd]ISPN100003: Finished local rebalance
> [transactions-repository-2-1gk60] 18:26:46,663 INFO [org.infinispan.CLUSTER] (transport-thread--p4-t22) [Context=memcachedCache][Context=transactions-repository-2-1gk60]ISPN100003: Finished local rebalance
> [transactions-repository-2-1gk60] 18:26:46,664 INFO [org.infinispan.CLUSTER] (transport-thread--p4-t22) ISPN000336: Finished cluster-wide rebalance for cache memcachedCache, topology id = 21
> [transactions-repository-2-pd4c7] 18:26:46,668 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t56) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,668 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t65) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-jms3k] 18:26:46,673 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-4) DGISPN0002: Stopped memcachedCache cache from clustered container
> [transactions-repository-2-pd4c7] 18:26:46,674 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t56) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,675 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t65) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,679 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t56) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,679 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t65) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,686 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t65) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,687 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t56) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,713 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t56) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,715 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t65) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,722 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,722 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t56) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,728 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,728 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t56) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,729 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t56) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,729 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,747 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t56) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,748 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,753 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t56) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,753 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,755 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t56) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,756 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,757 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t56) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,758 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,759 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t56) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,760 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,760 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,760 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t56) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,763 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t64) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,765 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t62) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,765 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t65) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,766 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t58) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,766 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t63) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,765 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t53) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,763 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t56) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,764 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,774 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t53) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,774 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t56) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,777 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t56) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,777 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t53) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,778 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t53) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,778 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,778 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t63) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-jms3k] 18:26:46,778 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-2) DGISPN0002: Stopped transactions cache from clustered container
> [transactions-repository-2-pd4c7] 18:26:46,778 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t56) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,784 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t63) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,784 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t56) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,787 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t63) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,787 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t56) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-1gk60] 18:26:46,802 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 6 of cache transactions
> {noformat}
> Full logs from Rolling Update process might be found here: https://gist.github.com/slaskawi/d789431ab5f16d46136dafffd7d4a89e
> Steps to reproduce:
> * Start local OpenShift Cluster
> * invoke `./init_infrastructure.sh` from https://github.com/slaskawi/presentations/tree/ISPN-7487-reproducer
> * invoke `cd transaction-creator && mvn fabric8:run`
> * Start Spring Session Demo `cd session-demo && mvn fabric8:run`
> * Create a client which inserts data (`watch -n 0.5 curl http://<spring-session-demo-pod-ip>/sessions`) and at the same time invoke the rolling update: `oc deploy transactions-repository --latest -n myproject`
> * Observe logs `kubetail -l environment=infrastructure`
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
9 years, 1 month
- 1
- 0
[JBoss JIRA] (ISPN-7491) When a node leaves the cluster it emits unsolicited state
by Sebastian Łaskawiec (JIRA)
[ https://issues.jboss.org/browse/ISPN-7491?page=com.atlassian.jira.plugin.... ]
Sebastian Łaskawiec commented on ISPN-7491:
-------------------------------------------
I've been talking about this with [~dan.berindei]. Here's the most important bit:
{quote}
Feb 20 15:54:24 <dberindei> slaskawi: it looks to me as if node transactions-repository-2-pd4c7 requested some state from transactions-repository-2-jms3k, but transactions-repository-2-jms3k left while it was sending state, so some state arrived on pd4c7 after the topology update from the coordinator that didn't include jms3k
Feb 20 15:54:31 <dberindei> slaskawi: so pretty harmless
{quote}
> When a node leaves the cluster it emits unsolicited state
> ---------------------------------------------------------
>
> Key: ISPN-7491
> URL: https://issues.jboss.org/browse/ISPN-7491
> Project: Infinispan
> Issue Type: Bug
> Components: Cloud Integrations, Core
> Affects Versions: 9.0.0.CR1
> Environment: * OpenShift {{v1.5.0-alpha.2+e4b43ee}}
> * Custom Infinispan Server build (based on [these instructions|https://github.com/slaskawi/infinispan-1/tree/custom_image]). SHA1 {{2b0731b21649a88a75ed71d21b9cc06ba365e947}}
> Reporter: Sebastian Łaskawiec
> Priority: Blocker
>
> When I was performing [Spring Session and Kubernetes Rolling Update demo|https://bluejeans.com/s/pYKUg/] I encountered a couple of problems.
> One of the is this:
> {noformat}
> [transactions-repository-2-1gk60] 18:26:44,264 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t22) ISPN000336: Finished cluster-wide rebalance for cache ___protobuf_metadata, topology id = 18
> [transactions-repository-2-pd4c7] [GC (Allocation Failure) 756168K->489129K(1013632K), 0.0439983 secs]
> [transactions-repository-2-1gk60] [GC (Allocation Failure) 686020K->416468K(1013632K), 0.0514641 secs]
> [transactions-repository-2-pd4c7] [GC (Allocation Failure) 768745K->501140K(1013632K), 0.0544712 secs]
> [transactions-repository-2-1gk60] [GC (Allocation Failure) 696084K->427206K(1013632K), 0.0997416 secs]
> [transactions-repository-2-pd4c7] [GC (Allocation Failure) 780756K->516161K(1013632K), 0.0674455 secs]
> [transactions-repository-2-jms3k] *** JBossAS process (81) received TERM signal ***
> [transactions-repository-2-jms3k] 18:26:46,371 INFO [org.jboss.as.server] (Thread-2) WFLYSRV0220: Server shutdown has been requested via an OS signal
> [transactions-repository-2-jms3k] 18:26:46,392 INFO [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-5) WFLYJCA0010: Unbound data source [java:jboss/datasources/ExampleDS]
> [transactions-repository-2-1gk60] 18:26:46,395 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t25) ISPN000310: Starting cluster-wide rebalance for cache sessions, topology CacheTopology{id=25, rebalanceId=9, currentCH=DefaultConsistentHash{ns=20, owners = (3)[transactions-repository-2-1gk60: 5+6, transactions-repository-2-pd4c7: 7+3, transactions-repository-3-6mzvd: 8+3]}, pendingCH=DefaultConsistentHash{ns=20, owners = (3)[transactions-repository-2-1gk60: 7+8, transactions-repository-2-pd4c7: 7+6, transactions-repository-3-6mzvd: 6+6]}, unionCH=null, actualMembers=[transactions-repository-2-1gk60, transactions-repository-2-pd4c7, transactions-repository-3-6mzvd], persistentUUIDs=[d47f6f69-97e8-4385-abd0-f1d43312d8c4, a17aa2e7-5137-4b5b-bf90-266207db7bb2, a000aa28-b1bd-453b-86e9-1628da898c36]}
> [transactions-repository-2-1gk60] 18:26:46,395 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t25) [Context=sessions][Context=transactions-repository-2-1gk60]ISPN100002: Started local rebalance
> [transactions-repository-2-pd4c7] 18:26:46,402 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t53) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,411 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t24) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,417 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t53) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-jms3k] 18:26:46,430 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-6) DGISPN0002: Stopped sessions cache from clustered container
> [transactions-repository-2-pd4c7] 18:26:46,441 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t24) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,444 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t24) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,444 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t54) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,448 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t54) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,450 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t54) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,450 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t55) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,451 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t54) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,455 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t54) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,456 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t24) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,453 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t55) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,466 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t55) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,470 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t55) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,471 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t58) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,466 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t54) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,473 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t56) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,495 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,497 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t63) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,473 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t60) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,494 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t61) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,490 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t62) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,477 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t64) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,476 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t24) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,476 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t55) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,475 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t57) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,475 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t59) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,475 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t54) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,474 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t58) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-1gk60] 18:26:46,513 INFO [org.infinispan.CLUSTER] (transport-thread--p4-t6) [Context=sessions][Context=transactions-repository-2-1gk60]ISPN100003: Finished local rebalance
> [transactions-repository-2-pd4c7] 18:26:46,514 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t57) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,514 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t60) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,515 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t60) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,515 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t55) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,515 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,515 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t58) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,516 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t56) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,516 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t62) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,516 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t55) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-1gk60] 18:26:46,527 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t14) [Context=sessions][Context=transactions-repository-2-pd4c7]ISPN100003: Finished local rebalance
> [transactions-repository-2-pd4c7] 18:26:46,517 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t64) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,517 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-jms3k] 18:26:46,529 INFO [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-8) WFLYJCA0019: Stopped Driver service with driver-name = h2
> [transactions-repository-2-pd4c7] 18:26:46,536 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,541 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t65) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-1gk60] 18:26:46,548 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t19) [Context=sessions][Context=transactions-repository-3-6mzvd]ISPN100003: Finished local rebalance
> [transactions-repository-2-1gk60] 18:26:46,549 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t19) ISPN000336: Finished cluster-wide rebalance for cache sessions, topology id = 25
> [transactions-repository-2-pd4c7] 18:26:46,558 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t53) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,558 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-1gk60] 18:26:46,574 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t10) ISPN000310: Starting cluster-wide rebalance for cache default, topology CacheTopology{id=21, rebalanceId=10, currentCH=DefaultConsistentHash{ns=20, owners = (3)[transactions-repository-2-1gk60: 5+6, transactions-repository-2-pd4c7: 7+3, transactions-repository-3-6mzvd: 8+3]}, pendingCH=DefaultConsistentHash{ns=20, owners = (3)[transactions-repository-2-1gk60: 7+8, transactions-repository-2-pd4c7: 7+6, transactions-repository-3-6mzvd: 6+6]}, unionCH=null, actualMembers=[transactions-repository-2-1gk60, transactions-repository-2-pd4c7, transactions-repository-3-6mzvd], persistentUUIDs=[d47f6f69-97e8-4385-abd0-f1d43312d8c4, a17aa2e7-5137-4b5b-bf90-266207db7bb2, a000aa28-b1bd-453b-86e9-1628da898c36]}
> [transactions-repository-2-1gk60] 18:26:46,574 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t10) [Context=default][Context=transactions-repository-2-1gk60]ISPN100002: Started local rebalance
> [transactions-repository-2-pd4c7] 18:26:46,536 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t64) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,563 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,517 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t58) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,563 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t53) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,517 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t63) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-1gk60] 18:26:46,591 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t2) ISPN000310: Starting cluster-wide rebalance for cache memcachedCache, topology CacheTopology{id=21, rebalanceId=10, currentCH=DefaultConsistentHash{ns=20, owners = (3)[transactions-repository-2-1gk60: 5+6, transactions-repository-2-pd4c7: 7+3, transactions-repository-3-6mzvd: 8+3]}, pendingCH=DefaultConsistentHash{ns=20, owners = (3)[transactions-repository-2-1gk60: 7+8, transactions-repository-2-pd4c7: 7+6, transactions-repository-3-6mzvd: 6+6]}, unionCH=null, actualMembers=[transactions-repository-2-1gk60, transactions-repository-2-pd4c7, transactions-repository-3-6mzvd], persistentUUIDs=[d47f6f69-97e8-4385-abd0-f1d43312d8c4, a17aa2e7-5137-4b5b-bf90-266207db7bb2, a000aa28-b1bd-453b-86e9-1628da898c36]}
> [transactions-repository-2-1gk60] 18:26:46,596 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t2) [Context=memcachedCache][Context=transactions-repository-2-1gk60]ISPN100002: Started local rebalance
> [transactions-repository-2-1gk60] 18:26:46,602 INFO [org.infinispan.CLUSTER] (transport-thread--p4-t25) [Context=default][Context=transactions-repository-2-1gk60]ISPN100003: Finished local rebalance
> [transactions-repository-2-jms3k] 18:26:46,599 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-7) DGISPN0002: Stopped default cache from clustered container
> [transactions-repository-2-pd4c7] 18:26:46,609 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t58) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,610 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,610 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t63) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,610 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t64) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,611 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,611 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t64) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,611 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,611 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t58) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,612 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t55) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,611 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t64) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,611 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t63) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,612 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-1gk60] 18:26:46,635 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t6) [Context=default][Context=transactions-repository-2-pd4c7]ISPN100003: Finished local rebalance
> [transactions-repository-2-1gk60] 18:26:46,640 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t15) [Context=default][Context=transactions-repository-3-6mzvd]ISPN100003: Finished local rebalance
> [transactions-repository-2-1gk60] 18:26:46,642 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t15) ISPN000336: Finished cluster-wide rebalance for cache default, topology id = 21
> [transactions-repository-2-1gk60] 18:26:46,644 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t20) [Context=memcachedCache][Context=transactions-repository-2-pd4c7]ISPN100003: Finished local rebalance
> [transactions-repository-2-pd4c7] 18:26:46,641 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t64) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,641 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t55) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,641 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,641 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t53) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,641 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t63) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,641 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t58) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,641 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,641 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t65) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,641 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t64) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,641 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t62) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-1gk60] 18:26:46,661 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t9) [Context=memcachedCache][Context=transactions-repository-3-6mzvd]ISPN100003: Finished local rebalance
> [transactions-repository-2-1gk60] 18:26:46,663 INFO [org.infinispan.CLUSTER] (transport-thread--p4-t22) [Context=memcachedCache][Context=transactions-repository-2-1gk60]ISPN100003: Finished local rebalance
> [transactions-repository-2-1gk60] 18:26:46,664 INFO [org.infinispan.CLUSTER] (transport-thread--p4-t22) ISPN000336: Finished cluster-wide rebalance for cache memcachedCache, topology id = 21
> [transactions-repository-2-pd4c7] 18:26:46,668 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t56) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,668 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t65) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-jms3k] 18:26:46,673 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-4) DGISPN0002: Stopped memcachedCache cache from clustered container
> [transactions-repository-2-pd4c7] 18:26:46,674 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t56) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,675 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t65) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,679 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t56) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,679 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t65) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,686 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t65) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,687 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t56) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,713 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t56) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,715 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t65) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,722 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,722 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t56) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,728 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,728 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t56) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,729 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t56) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,729 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,747 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t56) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,748 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,753 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t56) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,753 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,755 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t56) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,756 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,757 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t56) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,758 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,759 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t56) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,760 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,760 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,760 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t56) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,763 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t64) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,765 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t62) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,765 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t65) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,766 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t58) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,766 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t63) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,765 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t53) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,763 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t56) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,764 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,774 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t53) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,774 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t56) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,777 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t56) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,777 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t53) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,778 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t53) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,778 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,778 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t63) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-jms3k] 18:26:46,778 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-2) DGISPN0002: Stopped transactions cache from clustered container
> [transactions-repository-2-pd4c7] 18:26:46,778 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t56) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,784 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t63) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,784 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t56) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,787 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t63) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 19 of cache transactions
> [transactions-repository-2-pd4c7] 18:26:46,787 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t56) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 17 of cache transactions
> [transactions-repository-2-1gk60] 18:26:46,802 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p7-t66) ISPN000396: Received unsolicited state from node transactions-repository-2-jms3k for segment 6 of cache transactions
> {noformat}
> Full logs from Rolling Update process might be found here: https://gist.github.com/slaskawi/d789431ab5f16d46136dafffd7d4a89e
> Steps to reproduce:
> * Start local OpenShift Cluster
> * invoke `./init_infrastructure.sh` from https://github.com/slaskawi/presentations/tree/ISPN-7487-reproducer
> * invoke `cd transaction-creator && mvn fabric8:run`
> * Start Spring Session Demo `cd session-demo && mvn fabric8:run`
> * Create a client which inserts data (`watch -n 0.5 curl http://<spring-session-demo-pod-ip>/sessions`) and at the same time invoke the rolling update: `oc deploy transactions-repository --latest -n myproject`
> * Observe logs `kubetail -l environment=infrastructure`
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
9 years, 1 month
- 1
- 0
[JBoss JIRA] (ISPN-7496) InitialStateTransfer should always retry
by Sebastian Łaskawiec (JIRA)
[ https://issues.jboss.org/browse/ISPN-7496?page=com.atlassian.jira.plugin.... ]
Sebastian Łaskawiec updated ISPN-7496:
--------------------------------------
Status: Open (was: New)
> InitialStateTransfer should always retry
> ----------------------------------------
>
> Key: ISPN-7496
> URL: https://issues.jboss.org/browse/ISPN-7496
> Project: Infinispan
> Issue Type: Bug
> Components: Cloud Integrations
> Affects Versions: 9.0.0.Beta2
> Environment: * OpenShift {{v1.5.0-alpha.2+e4b43ee}}
> * Infinispan Server 9.0.0.Beta2
> Reporter: Sebastian Łaskawiec
> Assignee: Sebastian Łaskawiec
> Priority: Blocker
>
> When I was performing [Spring Session and Kubernetes Rolling Update demo|https://bluejeans.com/s/pYKUg/] I encountered a couple of problems.
> One of the is this:
> {noformat}
> 13:21:40,816 ERROR [org.jboss.msc.service.fail] (MSC service thread 1-6) MSC000001: Failed to start service jboss.datagrid-infinispan.clustered.transactions: org.jboss.msc.service.StartException in service jboss.datagrid-infinispan.clustered.transactions: Failed to start service
> at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1904)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.Exception on object of type StateTransferManagerImpl
> at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:172)
> at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:867)
> at org.infinispan.factories.AbstractComponentRegistry.invokeStartMethods(AbstractComponentRegistry.java:633)
> at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:622)
> at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:547)
> at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:231)
> at org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:803)
> at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:411)
> at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:641)
> at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:589)
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:453)
> at org.infinispan.manager.impl.AbstractDelegatingEmbeddedCacheManager.getCache(AbstractDelegatingEmbeddedCacheManager.java:139)
> at org.infinispan.server.infinispan.SecurityActions$5.run(SecurityActions.java:133)
> at org.infinispan.server.infinispan.SecurityActions$5.run(SecurityActions.java:130)
> at org.infinispan.security.Security.doPrivileged(Security.java:76)
> at org.infinispan.server.infinispan.SecurityActions.doPrivileged(SecurityActions.java:66)
> at org.infinispan.server.infinispan.SecurityActions.startCache(SecurityActions.java:138)
> at org.jboss.as.clustering.infinispan.subsystem.CacheService.start(CacheService.java:86)
> at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1948)
> at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1881)
> ... 3 more
> Caused by: org.infinispan.commons.CacheException: Initial state transfer timed out for cache transactions on transactions-repository-2-m3rqk
> at org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete(StateTransferManagerImpl.java:228)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:168)
> ... 22 more
> {noformat}
> Full logs from Rolling Update process might be found here: https://gist.github.com/slaskawi/92b9fa309b1e07cbfdc7353db45ed754#file-gi...
> Steps to reproduce:
> * Start local OpenShift Cluster
> * invoke `./init_infrastructure.sh` from https://github.com/slaskawi/presentations/tree/ISPN-7487-reproducer
> * invoke `cd transaction-creator && mvn fabric8:run`
> * Observe logs `kubetail -l environment=infrastructure`
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
9 years, 1 month
- 1
- 0
[JBoss JIRA] (ISPN-7496) InitialStateTransfer should always retry
by Sebastian Łaskawiec (JIRA)
[ https://issues.jboss.org/browse/ISPN-7496?page=com.atlassian.jira.plugin.... ]
Sebastian Łaskawiec updated ISPN-7496:
--------------------------------------
Status: Pull Request Sent (was: Open)
Git Pull Request: https://github.com/infinispan/infinispan/pull/4869
> InitialStateTransfer should always retry
> ----------------------------------------
>
> Key: ISPN-7496
> URL: https://issues.jboss.org/browse/ISPN-7496
> Project: Infinispan
> Issue Type: Bug
> Components: Cloud Integrations
> Affects Versions: 9.0.0.Beta2
> Environment: * OpenShift {{v1.5.0-alpha.2+e4b43ee}}
> * Infinispan Server 9.0.0.Beta2
> Reporter: Sebastian Łaskawiec
> Assignee: Sebastian Łaskawiec
> Priority: Blocker
>
> When I was performing [Spring Session and Kubernetes Rolling Update demo|https://bluejeans.com/s/pYKUg/] I encountered a couple of problems.
> One of the is this:
> {noformat}
> 13:21:40,816 ERROR [org.jboss.msc.service.fail] (MSC service thread 1-6) MSC000001: Failed to start service jboss.datagrid-infinispan.clustered.transactions: org.jboss.msc.service.StartException in service jboss.datagrid-infinispan.clustered.transactions: Failed to start service
> at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1904)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.Exception on object of type StateTransferManagerImpl
> at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:172)
> at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:867)
> at org.infinispan.factories.AbstractComponentRegistry.invokeStartMethods(AbstractComponentRegistry.java:633)
> at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:622)
> at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:547)
> at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:231)
> at org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:803)
> at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:411)
> at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:641)
> at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:589)
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:453)
> at org.infinispan.manager.impl.AbstractDelegatingEmbeddedCacheManager.getCache(AbstractDelegatingEmbeddedCacheManager.java:139)
> at org.infinispan.server.infinispan.SecurityActions$5.run(SecurityActions.java:133)
> at org.infinispan.server.infinispan.SecurityActions$5.run(SecurityActions.java:130)
> at org.infinispan.security.Security.doPrivileged(Security.java:76)
> at org.infinispan.server.infinispan.SecurityActions.doPrivileged(SecurityActions.java:66)
> at org.infinispan.server.infinispan.SecurityActions.startCache(SecurityActions.java:138)
> at org.jboss.as.clustering.infinispan.subsystem.CacheService.start(CacheService.java:86)
> at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1948)
> at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1881)
> ... 3 more
> Caused by: org.infinispan.commons.CacheException: Initial state transfer timed out for cache transactions on transactions-repository-2-m3rqk
> at org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete(StateTransferManagerImpl.java:228)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:168)
> ... 22 more
> {noformat}
> Full logs from Rolling Update process might be found here: https://gist.github.com/slaskawi/92b9fa309b1e07cbfdc7353db45ed754#file-gi...
> Steps to reproduce:
> * Start local OpenShift Cluster
> * invoke `./init_infrastructure.sh` from https://github.com/slaskawi/presentations/tree/ISPN-7487-reproducer
> * invoke `cd transaction-creator && mvn fabric8:run`
> * Observe logs `kubetail -l environment=infrastructure`
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
9 years, 1 month
- 1
- 0
[JBoss JIRA] (ISPN-7502) Upgrade to Hibernate Search 5.7.0.Final
by Sanne Grinovero (JIRA)
[ https://issues.jboss.org/browse/ISPN-7502?page=com.atlassian.jira.plugin.... ]
Sanne Grinovero updated ISPN-7502:
----------------------------------
Status: Pull Request Sent (was: Open)
Git Pull Request: https://github.com/infinispan/infinispan/pull/4874
> Upgrade to Hibernate Search 5.7.0.Final
> ---------------------------------------
>
> Key: ISPN-7502
> URL: https://issues.jboss.org/browse/ISPN-7502
> Project: Infinispan
> Issue Type: Component Upgrade
> Components: Embedded Querying
> Reporter: Sanne Grinovero
> Assignee: Sanne Grinovero
> Fix For: 9.0.0.CR2
>
>
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
9 years, 1 month
- 1
- 0
[JBoss JIRA] (ISPN-7502) Upgrade to Hibernate Search 5.7.0.Final
by Sanne Grinovero (JIRA)
[ https://issues.jboss.org/browse/ISPN-7502?page=com.atlassian.jira.plugin.... ]
Sanne Grinovero updated ISPN-7502:
----------------------------------
Status: Open (was: New)
> Upgrade to Hibernate Search 5.7.0.Final
> ---------------------------------------
>
> Key: ISPN-7502
> URL: https://issues.jboss.org/browse/ISPN-7502
> Project: Infinispan
> Issue Type: Component Upgrade
> Components: Embedded Querying
> Reporter: Sanne Grinovero
> Assignee: Sanne Grinovero
> Fix For: 9.0.0.CR2
>
>
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
9 years, 1 month
- 1
- 0
[JBoss JIRA] (ISPN-7502) Upgrade to Hibernate Search 5.7.0.Final
by Sanne Grinovero (JIRA)
Sanne Grinovero created ISPN-7502:
-------------------------------------
Summary: Upgrade to Hibernate Search 5.7.0.Final
Key: ISPN-7502
URL: https://issues.jboss.org/browse/ISPN-7502
Project: Infinispan
Issue Type: Component Upgrade
Components: Embedded Querying
Reporter: Sanne Grinovero
Assignee: Sanne Grinovero
Fix For: 9.0.0.CR2
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
9 years, 1 month
- 1
- 0
[JBoss JIRA] (ISPN-7135) ClusterListenerReplTest.testPrimaryOwnerGoesDownBeforeBackupRaisesEvent random failures
by William Burns (JIRA)
[ https://issues.jboss.org/browse/ISPN-7135?page=com.atlassian.jira.plugin.... ]
Work on ISPN-7135 started by William Burns.
-------------------------------------------
> ClusterListenerReplTest.testPrimaryOwnerGoesDownBeforeBackupRaisesEvent random failures
> ---------------------------------------------------------------------------------------
>
> Key: ISPN-7135
> URL: https://issues.jboss.org/browse/ISPN-7135
> Project: Infinispan
> Issue Type: Bug
> Components: Test Suite - Core
> Affects Versions: 9.0.0.Alpha4
> Reporter: Dan Berindei
> Assignee: William Burns
> Priority: Critical
> Labels: testsuite_stability
> Fix For: 9.0.0.CR2
>
>
> The test assumes that because the primary owner is killed before the backups updated the entry, the clustered listener will only receive one {{CACHE_ENTRY_CREATED}} event.
> However, the crash and the following rebalance require 3 topology updates, meaning the command could be retried a second time, after the backup already applied the new value. In the log below, {{NodeDS}} retries with topology {{7}}, the primary {{NodeDU}} forwards it to the backup (also {{NodeDS}}), but then sees topology {{8}} and triggers a retry. The clustered listener on {{NodeDS}} receives both a {{CACHE_ENTRY_CREATED}} event and a {{CACHE_ENTRY_MODIFIED}} event.
> {noformat}
> 10:35:47,429 TRACE (ForkThread-2,Test:[]) [BaseDistributionInterceptor] Command topology id is 5, current topology id is 5, successful? true
> 10:35:47,429 TRACE (ForkThread-2,Test:[]) [BaseDistributionInterceptor] I'm not the primary owner, so sending the command to the primary owner(Test-NodeDT-11194) in order to be forwarded
> 10:35:47,496 DEBUG (remote-thread-Test-NodeDS-p31418-t6:[cluster-listener]) [ClusterTopologyManagerImpl] Updating cluster-wide current topology for cache cluster-listener, topology = CacheTopology{id=6, rebalanceId=3, currentCH=ReplicatedConsistentHash{ns = 256, owners = (2)[Test-NodeDS-11906: 133, Test-NodeDU-38945: 123]}, pendingCH=null, unionCH=null, actualMembers=[Test-NodeDS-11906, Test-NodeDU-38945], persistentUUIDs=[7ddc513a-2a94-4dc5-9918-7fb90a6510bb, cd827dd6-62d9-4f78-a28e-4d10dc77a862]}, availability mode = AVAILABLE
> 10:35:47,502 TRACE (transport-thread-Test-NodeDS-p31420-t1:[Topology-cluster-listener]) [StateConsumerImpl] Received new topology for cache cluster-listener, isRebalance = false, isMember = true, topology = CacheTopology{id=6, rebalanceId=3, currentCH=PartitionerConsistentHash:ReplicatedConsistentHash{ns = 256, owners = (2)[Test-NodeDS-11906: 133, Test-NodeDU-38945: 123]}, pendingCH=null, unionCH=null, actualMembers=[Test-NodeDS-11906, Test-NodeDU-38945], persistentUUIDs=[7ddc513a-2a94-4dc5-9918-7fb90a6510bb, cd827dd6-62d9-4f78-a28e-4d10dc77a862]}
> 10:35:47,514 INFO (remote-thread-Test-NodeDS-p31418-t6:[cluster-listener]) [CLUSTER] ISPN000310: Starting cluster-wide rebalance for cache cluster-listener, topology CacheTopology{id=7, rebalanceId=4, currentCH=ReplicatedConsistentHash{ns = 256, owners = (2)[Test-NodeDS-11906: 133, Test-NodeDU-38945: 123]}, pendingCH=ReplicatedConsistentHash{ns = 256, owners = (2)[Test-NodeDS-11906: 134, Test-NodeDU-38945: 122]}, unionCH=null, actualMembers=[Test-NodeDS-11906, Test-NodeDU-38945], persistentUUIDs=[7ddc513a-2a94-4dc5-9918-7fb90a6510bb, cd827dd6-62d9-4f78-a28e-4d10dc77a862]}
> 10:35:47,526 TRACE (transport-thread-Test-NodeDS-p31420-t2:[Topology-cluster-listener]) [StateConsumerImpl] Received new topology for cache cluster-listener, isRebalance = true, isMember = true, topology = CacheTopology{id=7, rebalanceId=4, currentCH=PartitionerConsistentHash:ReplicatedConsistentHash{ns = 256, owners = (2)[Test-NodeDS-11906: 133, Test-NodeDU-38945: 123]}, pendingCH=PartitionerConsistentHash:ReplicatedConsistentHash{ns = 256, owners = (2)[Test-NodeDS-11906: 134, Test-NodeDU-38945: 122]}, unionCH=PartitionerConsistentHash:ReplicatedConsistentHash{ns = 256, owners = (2)[Test-NodeDS-11906: 133, Test-NodeDU-38945: 123]}, actualMembers=[Test-NodeDS-11906, Test-NodeDU-38945], persistentUUIDs=[7ddc513a-2a94-4dc5-9918-7fb90a6510bb, cd827dd6-62d9-4f78-a28e-4d10dc77a862]}
> 10:35:47,527 TRACE (transport-thread-Test-NodeDU-p31480-t5:[Topology-cluster-listener]) [StateConsumerImpl] Received new topology for cache cluster-listener, isRebalance = false, isMember = true, topology = CacheTopology{id=6, rebalanceId=3, currentCH=PartitionerConsistentHash:ReplicatedConsistentHash{ns = 256, owners = (2)[Test-NodeDS-11906: 133, Test-NodeDU-38945: 123]}, pendingCH=null, unionCH=null, actualMembers=[Test-NodeDS-11906, Test-NodeDU-38945], persistentUUIDs=[7ddc513a-2a94-4dc5-9918-7fb90a6510bb, cd827dd6-62d9-4f78-a28e-4d10dc77a862]}
> *** 10:35:47,555 TRACE (remote-thread-Test-NodeDS-p31418-t6:[]) [StateTransferInterceptor] Retrying command because of topology change, current topology is 7: PutKeyValueCommand{key=MagicKey{1088/5C01D0B5/73@Test-NodeDT-11194}, value=first-value, flags=[], putIfAbsent=false, valueMatcher=MATCH_ALWAYS, metadata=EmbeddedExpirableMetadata{lifespan=-1, maxIdle=-1, version=null}, successful=true}
> 10:35:47,555 TRACE (remote-thread-Test-NodeDS-p31418-t6:[]) [BaseDistributionInterceptor] I'm not the primary owner, so sending the command to the primary owner(Test-NodeDU-38945) in order to be forwarded
> 10:35:47,558 TRACE (transport-thread-Test-NodeDU-p31480-t5:[Topology-cluster-listener]) [StateConsumerImpl] Received new topology for cache cluster-listener, isRebalance = true, isMember = true, topology = CacheTopology{id=7, rebalanceId=4, currentCH=PartitionerConsistentHash:ReplicatedConsistentHash{ns = 256, owners = (2)[Test-NodeDS-11906: 133, Test-NodeDU-38945: 123]}, pendingCH=PartitionerConsistentHash:ReplicatedConsistentHash{ns = 256, owners = (2)[Test-NodeDS-11906: 134, Test-NodeDU-38945: 122]}, unionCH=PartitionerConsistentHash:ReplicatedConsistentHash{ns = 256, owners = (2)[Test-NodeDS-11906: 133, Test-NodeDU-38945: 123]}, actualMembers=[Test-NodeDS-11906, Test-NodeDU-38945], persistentUUIDs=[7ddc513a-2a94-4dc5-9918-7fb90a6510bb, cd827dd6-62d9-4f78-a28e-4d10dc77a862]}
> 10:35:47,600 DEBUG (remote-thread-Test-NodeDS-p31418-t5:[cluster-listener]) [ClusterTopologyManagerImpl] Updating cluster-wide current topology for cache cluster-listener, topology = CacheTopology{id=8, rebalanceId=4, currentCH=ReplicatedConsistentHash{ns = 256, owners = (2)[Test-NodeDS-11906: 134, Test-NodeDU-38945: 122]}, pendingCH=null, unionCH=null, actualMembers=[Test-NodeDS-11906, Test-NodeDU-38945], persistentUUIDs=[7ddc513a-2a94-4dc5-9918-7fb90a6510bb, cd827dd6-62d9-4f78-a28e-4d10dc77a862]}, availability mode = AVAILABLE
> 10:35:47,603 TRACE (remote-thread-Test-NodeDS-p31418-t6:[]) [ReadCommittedEntry] Updating entry (key=MagicKey{1088/5C01D0B5/73@Test-NodeDT-11194} removed=false valid=true changed=true created=true value=first-value metadata=EmbeddedExpirableMetadata{lifespan=-1, maxIdle=-1, version=null}, providedMetadata=EmbeddedExpirableMetadata{lifespan=-1, maxIdle=-1, version=null})
> 10:35:47,603 TRACE (transport-thread-Test-NodeDS-p31420-t6:[Topology-cluster-listener]) [StateConsumerImpl] Received new topology for cache cluster-listener, isRebalance = false, isMember = true, topology = CacheTopology{id=8, rebalanceId=4, currentCH=PartitionerConsistentHash:ReplicatedConsistentHash{ns = 256, owners = (2)[Test-NodeDS-11906: 134, Test-NodeDU-38945: 122]}, pendingCH=null, unionCH=null, actualMembers=[Test-NodeDS-11906, Test-NodeDU-38945], persistentUUIDs=[7ddc513a-2a94-4dc5-9918-7fb90a6510bb, cd827dd6-62d9-4f78-a28e-4d10dc77a862]}
> 10:35:47,611 DEBUG (remote-thread-Test-NodeDS-p31418-t6:[]) [Test] Adding new cluster event EventImpl{type=CACHE_ENTRY_CREATED, pre=false, cache=Cache 'cluster-listener'@Test-NodeDS-11906, key=MagicKey{1088/5C01D0B5/73@Test-NodeDT-11194}, value=first-value, oldValue=null, transaction=null, originLocal=false, transactionSuccessful=false, entries=null, created=false}
> 10:35:47,614 TRACE (transport-thread-Test-NodeDU-p31480-t1:[Topology-cluster-listener]) [StateConsumerImpl] Received new topology for cache cluster-listener, isRebalance = false, isMember = true, topology = CacheTopology{id=8, rebalanceId=4, currentCH=PartitionerConsistentHash:ReplicatedConsistentHash{ns = 256, owners = (2)[Test-NodeDS-11906: 134, Test-NodeDU-38945: 122]}, pendingCH=null, unionCH=null, actualMembers=[Test-NodeDS-11906, Test-NodeDU-38945], persistentUUIDs=[7ddc513a-2a94-4dc5-9918-7fb90a6510bb, cd827dd6-62d9-4f78-a28e-4d10dc77a862]}
> 10:35:47,660 TRACE (OOB-1,Test-NodeDS-11906:[]) [AsyncInvocationStage] Exception from invocation handler
> org.infinispan.statetransfer.OutdatedTopologyException: Cache topology changed while the command was executing: expected 7, got 8
> *** 10:35:47,661 TRACE (OOB-1,Test-NodeDS-11906:[]) [StateTransferInterceptor] Retrying command because of topology change, current topology is 8: PutKeyValueCommand{key=MagicKey{1088/5C01D0B5/73@Test-NodeDT-11194}, value=first-value, flags=[COMMAND_RETRY], putIfAbsent=false, valueMatcher=MATCH_ALWAYS, metadata=EmbeddedExpirableMetadata{lifespan=-1, maxIdle=-1, version=null}, successful=true}
> 10:35:47,661 TRACE (OOB-1,Test-NodeDS-11906:[]) [BaseDistributionInterceptor] I'm not the primary owner, so sending the command to the primary owner(Test-NodeDU-38945) in order to be forwarded
> 10:35:47,745 TRACE (remote-thread-Test-NodeDS-p31418-t6:[]) [ReadCommittedEntry] Updating entry (key=MagicKey{1088/5C01D0B5/73@Test-NodeDT-11194} removed=false valid=true changed=true created=false value=first-value metadata=EmbeddedExpirableMetadata{lifespan=-1, maxIdle=-1, version=null}, providedMetadata=EmbeddedExpirableMetadata{lifespan=-1, maxIdle=-1, version=null})
> 10:35:47,746 DEBUG (remote-thread-Test-NodeDS-p31418-t6:[]) [Test] Adding new cluster event EventImpl{type=CACHE_ENTRY_MODIFIED, pre=false, cache=Cache 'cluster-listener'@Test-NodeDS-11906, key=MagicKey{1088/5C01D0B5/73@Test-NodeDT-11194}, value=first-value, oldValue=first-value, transaction=null, originLocal=false, transactionSuccessful=false, entries=null, created=false}
> 10:35:47,771 TRACE (OOB-2,Test-NodeDU-38945:[]) [ReadCommittedEntry] Updating entry (key=MagicKey{1088/5C01D0B5/73@Test-NodeDT-11194} removed=false valid=true changed=true created=true value=first-value metadata=EmbeddedExpirableMetadata{lifespan=-1, maxIdle=-1, version=null}, providedMetadata=EmbeddedExpirableMetadata{lifespan=-1, maxIdle=-1, version=null})
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
9 years, 1 month
- 1
- 0