[infinispan-issues] [JBoss JIRA] (ISPN-7488) Fix NPE in ClusterCacheStatus.java:147

[ https://issues.jboss.org/browse/ISPN-7488?page=com.atlassian.jira.plugin.... ] Sebastian Łaskawiec updated ISPN-7488: -------------------------------------- Description: When I was performing [Spring Session and Kubernetes Rolling Update demo|https://bluejeans.com/s/pYKUg/] I encountered a couple of problems. One of the is this: {noformat} [transactions-repository-1-4z05w] 18:09:06,122 WARN [org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler] (remote-thread--p2-t5) ISPN000071: Caught exception when handling command CacheTopologyControlCommand{cache=___script_cache, type=REBALANCE_CONFIRM, sender=transactions-repository-1-1f8dx, joinInfo=null, topologyId=9, rebalanceId=0, currentCH=null, pendingCH=null, availabilityMode=null, actualMembers=null, throwable=null, viewId=4}: org.infinispan.commons.CacheConfigurationException: Unable to configure component (type: class org.infinispan.topology.CacheTopologyControlCommand, instance CacheTopologyControlCommand{cache=___script_cache, type=REBALANCE_CONFIRM, sender=transactions-repository-1-1f8dx, joinInfo=null, topologyId=9, rebalanceId=0, currentCH=null, pendingCH=null, availabilityMode=null, actualMembers=null, throwable=null, viewId=4}) [transactions-repository-1-4z05w] at org.infinispan.factories.AbstractComponentRegistry.wireDependencies(AbstractComponentRegistry.java:152) [transactions-repository-1-4z05w] at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler$2.run(GlobalInboundInvocationHandler.java:160) [transactions-repository-1-4z05w] at org.infinispan.util.concurrent.BlockingTaskAwareExecutorServiceImpl$RunnableWrapper.run(BlockingTaskAwareExecutorServiceImpl.java:203) [transactions-repository-1-4z05w] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [transactions-repository-1-4z05w] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [transactions-repository-1-4z05w] at java.lang.Thread.run(Thread.java:745) [transactions-repository-1-4z05w] Caused by: org.infinispan.IllegalLifecycleStateException: Trying to register a component after stopping: org.infinispan.topology.LocalTopologyManagerFactory [transactions-repository-1-4z05w] at org.infinispan.factories.AbstractComponentRegistry.registerComponentInternal(AbstractComponentRegistry.java:185) [transactions-repository-1-4z05w] at org.infinispan.factories.AbstractComponentRegistry.registerComponent(AbstractComponentRegistry.java:172) [transactions-repository-1-4z05w] at org.infinispan.factories.AbstractComponentRegistry.registerComponent(AbstractComponentRegistry.java:168) [transactions-repository-1-4z05w] at org.infinispan.factories.AbstractComponentRegistry.createComponentFactoryInternal(AbstractComponentRegistry.java:349) [transactions-repository-1-4z05w] at org.infinispan.factories.AbstractComponentRegistry.getFactory(AbstractComponentRegistry.java:328) [transactions-repository-1-4z05w] at org.infinispan.factories.AbstractComponentRegistry.getOrCreateComponent(AbstractComponentRegistry.java:294) [transactions-repository-1-4z05w] at org.infinispan.factories.AbstractComponentRegistry.invokeInjectionMethod(AbstractComponentRegistry.java:247) [transactions-repository-1-4z05w] at org.infinispan.factories.AbstractComponentRegistry.wireDependencies(AbstractComponentRegistry.java:148) [transactions-repository-1-4z05w] ... 5 more {noformat} Full logs from Rolling Update process might be found here: https://gist.github.com/slaskawi/530241bb695f1f490bcb25eabaf9d676 Steps to reproduce: * Start local OpenShift Cluster * invoke `./init_infrastructure.sh` from https://github.com/slaskawi/presentations/tree/ISPN-7487-reproducer * invoke `cd transaction-creator && mvn fabric8:run` * Do the rolling update: `oc deploy transactions-repository --latest -n myproject` * Observe logs `kubetail -l environment=infrastructure` was: When I was performing [Spring Session and Kubernetes Rolling Update demo|https://bluejeans.com/s/pYKUg/] I encountered a couple of problems. One of the is this: {noformat} kubetail -l environment=infrastructure Will tail 3 logs... transactions-repository-1-04x09 transactions-repository-1-1f8dx transactions-repository-1-4z05w [transactions-repository-1-1f8dx] JAVA_OPTS already set in environment; overriding default settings with values: -server -Xms1G -Xmx1G -XX:+UseConcMarkSweepGC -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -XX:+PrintGC [transactions-repository-1-1f8dx] ========================================================================= [transactions-repository-1-1f8dx] [transactions-repository-1-1f8dx] JBoss Bootstrap Environment [transactions-repository-1-1f8dx] [transactions-repository-1-1f8dx] JBOSS_HOME: /opt/jboss/infinispan-server [transactions-repository-1-1f8dx] [transactions-repository-1-1f8dx] JAVA: /usr/lib/jvm/java/bin/java [transactions-repository-1-1f8dx] [transactions-repository-1-1f8dx] JAVA_OPTS: -server -Xms1G -Xmx1G -XX:+UseConcMarkSweepGC -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -XX:+PrintGC [transactions-repository-1-1f8dx] [transactions-repository-1-1f8dx] ========================================================================= [transactions-repository-1-1f8dx] [transactions-repository-1-1f8dx] 17:54:15,634 INFO [org.jboss.modules] (main) JBoss Modules version 1.5.2.Final [transactions-repository-1-1f8dx] 17:54:15,853 INFO [org.jboss.msc] (main) JBoss MSC version 1.2.6.Final [transactions-repository-1-1f8dx] 17:54:15,931 INFO [org.jboss.as] (MSC service thread 1-8) WFLYSRV0049: Infinispan Server 9.0.0-SNAPSHOT (WildFly Core 2.2.0.Final) starting [transactions-repository-1-1f8dx] 17:54:17,232 INFO [org.jboss.as.controller.management-deprecated] (ServerService Thread Pool -- 7) WFLYCTL0028: Attribute 'default-stack' in the resource at address '/subsystem=datagrid-jgroups' is deprecated, and may be removed in future version. See the attribute description in the output of the read-resource-description operation to learn more about the deprecation. [transactions-repository-1-1f8dx] [GC (CMS Initial Mark) 234913K(1013632K), 0.0473268 secs] [transactions-repository-1-1f8dx] 17:54:17,432 INFO [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0039: Creating http management service using socket-binding (management-http) [transactions-repository-1-1f8dx] 17:54:17,452 INFO [org.xnio] (MSC service thread 1-2) XNIO version 3.4.0.Final [transactions-repository-1-1f8dx] 17:54:17,464 INFO [org.xnio.nio] (MSC service thread 1-2) XNIO NIO Implementation Version 3.4.0.Final [transactions-repository-1-1f8dx] 17:54:17,516 INFO [org.jboss.remoting] (MSC service thread 1-3) JBoss Remoting version 4.0.21.Final [transactions-repository-1-1f8dx] 17:54:17,553 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 19) Activating Infinispan subsystem. [transactions-repository-1-1f8dx] [GC (GCLocker Initiated GC) 279722K->29726K(1013632K), 0.0377206 secs] [transactions-repository-1-1f8dx] 17:54:17,611 INFO [org.infinispan.server.jgroups] (ServerService Thread Pool -- 21) DGJGRP0001: Activating JGroups subsystem. [transactions-repository-1-1f8dx] 17:54:17,630 WARN [org.jboss.as.txn] (ServerService Thread Pool -- 29) WFLYTX0013: Node identifier property is set to the default value. Please make sure it is unique. [transactions-repository-1-1f8dx] 17:54:17,646 INFO [org.jboss.as.naming] (ServerService Thread Pool -- 26) WFLYNAM0001: Activating Naming Subsystem [transactions-repository-1-1f8dx] 17:54:17,646 INFO [org.jboss.as.security] (ServerService Thread Pool -- 28) WFLYSEC0002: Activating Security Subsystem [transactions-repository-1-1f8dx] 17:54:17,652 INFO [org.wildfly.extension.io] (ServerService Thread Pool -- 22) WFLYIO001: Worker 'default' has auto-configured to 16 core threads with 128 task threads based on your 8 available processors [transactions-repository-1-1f8dx] 17:54:17,653 INFO [org.jboss.as.security] (MSC service thread 1-8) WFLYSEC0001: Current PicketBox version=4.9.6.Final [transactions-repository-1-1f8dx] 17:54:17,673 INFO [org.jboss.as.connector] (MSC service thread 1-4) WFLYJCA0009: Starting JCA Subsystem (WildFly/IronJacamar 1.3.4.Final) [transactions-repository-1-1f8dx] 17:54:17,701 INFO [org.jboss.as.connector.subsystems.datasources] (ServerService Thread Pool -- 18) WFLYJCA0004: Deploying JDBC-compliant driver class org.h2.Driver (version 1.3) [transactions-repository-1-1f8dx] 17:54:17,703 INFO [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-6) WFLYJCA0018: Started Driver service with driver-name = h2 [transactions-repository-1-1f8dx] 17:54:17,744 INFO [org.jboss.as.naming] (MSC service thread 1-7) WFLYNAM0003: Starting Naming Service [transactions-repository-1-1f8dx] 17:54:18,268 INFO [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-5) WFLYJCA0001: Bound data source [java:jboss/datasources/ExampleDS] [transactions-repository-1-1f8dx] 17:54:18,425 WARN [org.infinispan.configuration.cache.IndexingConfigurationBuilder] (ServerService Thread Pool -- 19) ISPN000403: No indexable classes were defined for this indexed cache; switching to autodetection (support for autodetection will be removed in Infinispan 9.0). [transactions-repository-1-1f8dx] 17:54:18,429 INFO [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (ServerService Thread Pool -- 19) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be passivated. [transactions-repository-1-1f8dx] 17:54:18,779 INFO [org.jboss.as.server.deployment.scanner] (MSC service thread 1-5) WFLYDS0013: Started FileSystemDeploymentService for directory /opt/jboss/infinispan-server/standalone/deployments [transactions-repository-1-04x09] JAVA_OPTS already set in environment; overriding default settings with values: -server -Xms1G -Xmx1G -XX:+UseConcMarkSweepGC -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -XX:+PrintGC [transactions-repository-1-04x09] ========================================================================= [transactions-repository-1-04x09] [transactions-repository-1-04x09] JBoss Bootstrap Environment [transactions-repository-1-04x09] [transactions-repository-1-04x09] JBOSS_HOME: /opt/jboss/infinispan-server [transactions-repository-1-04x09] [transactions-repository-1-04x09] JAVA: /usr/lib/jvm/java/bin/java [transactions-repository-1-04x09] [transactions-repository-1-04x09] JAVA_OPTS: -server -Xms1G -Xmx1G -XX:+UseConcMarkSweepGC -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -XX:+PrintGC [transactions-repository-1-04x09] [transactions-repository-1-04x09] ========================================================================= [transactions-repository-1-04x09] [transactions-repository-1-04x09] 17:54:15,628 INFO [org.jboss.modules] (main) JBoss Modules version 1.5.2.Final [transactions-repository-1-04x09] 17:54:15,849 INFO [org.jboss.msc] (main) JBoss MSC version 1.2.6.Final [transactions-repository-1-04x09] 17:54:15,960 INFO [org.jboss.as] (MSC service thread 1-5) WFLYSRV0049: Infinispan Server 9.0.0-SNAPSHOT (WildFly Core 2.2.0.Final) starting [transactions-repository-1-04x09] [GC (CMS Initial Mark) 218119K(1013632K), 0.0624692 secs] [transactions-repository-1-04x09] 17:54:17,475 INFO [org.jboss.as.controller.management-deprecated] (ServerService Thread Pool -- 13) WFLYCTL0028: Attribute 'default-stack' in the resource at address '/subsystem=datagrid-jgroups' is deprecated, and may be removed in future version. See the attribute description in the output of the read-resource-description operation to learn more about the deprecation. [transactions-repository-1-04x09] 17:54:17,647 INFO [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0039: Creating http management service using socket-binding (management-http) [transactions-repository-1-04x09] 17:54:17,664 INFO [org.xnio] (MSC service thread 1-3) XNIO version 3.4.0.Final [transactions-repository-1-04x09] 17:54:17,683 INFO [org.xnio.nio] (MSC service thread 1-3) XNIO NIO Implementation Version 3.4.0.Final [transactions-repository-1-04x09] 17:54:17,763 INFO [org.jboss.remoting] (MSC service thread 1-4) JBoss Remoting version 4.0.21.Final [transactions-repository-1-04x09] [GC (Allocation Failure) 279616K->29652K(1013632K), 0.0690895 secs] [transactions-repository-1-04x09] 17:54:17,872 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 19) Activating Infinispan subsystem. [transactions-repository-1-04x09] 17:54:17,875 INFO [org.infinispan.server.jgroups] (ServerService Thread Pool -- 21) DGJGRP0001: Activating JGroups subsystem. [transactions-repository-1-04x09] [GC (CMS Final Remark) 68259K(1013632K), 0.0485831 secs] [transactions-repository-1-04x09] 17:54:17,933 INFO [org.wildfly.extension.io] (ServerService Thread Pool -- 22) WFLYIO001: Worker 'default' has auto-configured to 16 core threads with 128 task threads based on your 8 available processors [transactions-repository-1-04x09] 17:54:17,960 WARN [org.jboss.as.txn] (ServerService Thread Pool -- 29) WFLYTX0013: Node identifier property is set to the default value. Please make sure it is unique. [transactions-repository-1-04x09] 17:54:17,972 INFO [org.jboss.as.security] (ServerService Thread Pool -- 28) WFLYSEC0002: Activating Security Subsystem [transactions-repository-1-04x09] 17:54:17,972 INFO [org.jboss.as.naming] (ServerService Thread Pool -- 26) WFLYNAM0001: Activating Naming Subsystem [transactions-repository-1-04x09] 17:54:17,984 INFO [org.jboss.as.security] (MSC service thread 1-4) WFLYSEC0001: Current PicketBox version=4.9.6.Final [transactions-repository-1-04x09] 17:54:18,004 INFO [org.jboss.as.connector.subsystems.datasources] (ServerService Thread Pool -- 18) WFLYJCA0004: Deploying JDBC-compliant driver class org.h2.Driver (version 1.3) [transactions-repository-1-04x09] 17:54:18,027 INFO [org.jboss.as.naming] (MSC service thread 1-5) WFLYNAM0003: Starting Naming Service [transactions-repository-1-04x09] 17:54:18,053 INFO [org.jboss.as.connector] (MSC service thread 1-8) WFLYJCA0009: Starting JCA Subsystem (WildFly/IronJacamar 1.3.4.Final) [transactions-repository-1-04x09] 17:54:18,065 INFO [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-6) WFLYJCA0018: Started Driver service with driver-name = h2 [transactions-repository-1-04x09] 17:54:18,543 WARN [org.infinispan.configuration.cache.IndexingConfigurationBuilder] (ServerService Thread Pool -- 19) ISPN000403: No indexable classes were defined for this indexed cache; switching to autodetection (support for autodetection will be removed in Infinispan 9.0). [transactions-repository-1-04x09] 17:54:18,561 INFO [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (ServerService Thread Pool -- 19) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be passivated. [transactions-repository-1-04x09] 17:54:18,584 INFO [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-6) WFLYJCA0001: Bound data source [java:jboss/datasources/ExampleDS] [transactions-repository-1-04x09] 17:54:18,795 INFO [org.jboss.as.server.deployment.scanner] (MSC service thread 1-2) WFLYDS0013: Started FileSystemDeploymentService for directory /opt/jboss/infinispan-server/standalone/deployments [transactions-repository-1-4z05w] JAVA_OPTS already set in environment; overriding default settings with values: -server -Xms1G -Xmx1G -XX:+UseConcMarkSweepGC -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -XX:+PrintGC [transactions-repository-1-4z05w] ========================================================================= [transactions-repository-1-4z05w] [transactions-repository-1-4z05w] JBoss Bootstrap Environment [transactions-repository-1-4z05w] [transactions-repository-1-4z05w] JBOSS_HOME: /opt/jboss/infinispan-server [transactions-repository-1-4z05w] [transactions-repository-1-4z05w] JAVA: /usr/lib/jvm/java/bin/java [transactions-repository-1-4z05w] [transactions-repository-1-4z05w] JAVA_OPTS: -server -Xms1G -Xmx1G -XX:+UseConcMarkSweepGC -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -XX:+PrintGC [transactions-repository-1-4z05w] [transactions-repository-1-4z05w] ========================================================================= [transactions-repository-1-4z05w] [transactions-repository-1-4z05w] 17:54:15,714 INFO [org.jboss.modules] (main) JBoss Modules version 1.5.2.Final [transactions-repository-1-4z05w] 17:54:15,950 INFO [org.jboss.msc] (main) JBoss MSC version 1.2.6.Final [transactions-repository-1-4z05w] 17:54:16,016 INFO [org.jboss.as] (MSC service thread 1-6) WFLYSRV0049: Infinispan Server 9.0.0-SNAPSHOT (WildFly Core 2.2.0.Final) starting [transactions-repository-1-4z05w] 17:54:17,364 INFO [org.jboss.as.controller.management-deprecated] (ServerService Thread Pool -- 5) WFLYCTL0028: Attribute 'default-stack' in the resource at address '/subsystem=datagrid-jgroups' is deprecated, and may be removed in future version. See the attribute description in the output of the read-resource-description operation to learn more about the deprecation. [transactions-repository-1-4z05w] [GC (CMS Initial Mark) 234897K(1013632K), 0.0587177 secs] [transactions-repository-1-4z05w] 17:54:17,599 INFO [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0039: Creating http management service using socket-binding (management-http) [transactions-repository-1-4z05w] 17:54:17,620 INFO [org.xnio] (MSC service thread 1-5) XNIO version 3.4.0.Final [transactions-repository-1-4z05w] 17:54:17,626 INFO [org.xnio.nio] (MSC service thread 1-5) XNIO NIO Implementation Version 3.4.0.Final [transactions-repository-1-4z05w] 17:54:17,674 INFO [org.jboss.remoting] (MSC service thread 1-5) JBoss Remoting version 4.0.21.Final [transactions-repository-1-4z05w] [GC (Allocation Failure) 279616K->29736K(1013632K), 0.0733498 secs] [transactions-repository-1-4z05w] 17:54:17,799 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 19) Activating Infinispan subsystem. [transactions-repository-1-4z05w] 17:54:17,802 WARN [org.jboss.as.txn] (ServerService Thread Pool -- 29) WFLYTX0013: Node identifier property is set to the default value. Please make sure it is unique. [transactions-repository-1-4z05w] 17:54:17,829 INFO [org.jboss.as.security] (ServerService Thread Pool -- 28) WFLYSEC0002: Activating Security Subsystem [transactions-repository-1-4z05w] 17:54:17,831 INFO [org.infinispan.server.jgroups] (ServerService Thread Pool -- 21) DGJGRP0001: Activating JGroups subsystem. [transactions-repository-1-4z05w] 17:54:17,835 INFO [org.wildfly.extension.io] (ServerService Thread Pool -- 22) WFLYIO001: Worker 'default' has auto-configured to 16 core threads with 128 task threads based on your 8 available processors [transactions-repository-1-4z05w] 17:54:17,870 INFO [org.jboss.as.naming] (ServerService Thread Pool -- 26) WFLYNAM0001: Activating Naming Subsystem [transactions-repository-1-4z05w] 17:54:17,881 INFO [org.jboss.as.connector.subsystems.datasources] (ServerService Thread Pool -- 18) WFLYJCA0004: Deploying JDBC-compliant driver class org.h2.Driver (version 1.3) [transactions-repository-1-4z05w] 17:54:17,887 INFO [org.jboss.as.connector] (MSC service thread 1-2) WFLYJCA0009: Starting JCA Subsystem (WildFly/IronJacamar 1.3.4.Final) [transactions-repository-1-4z05w] 17:54:17,895 INFO [org.jboss.as.security] (MSC service thread 1-4) WFLYSEC0001: Current PicketBox version=4.9.6.Final [transactions-repository-1-4z05w] 17:54:17,900 INFO [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-8) WFLYJCA0018: Started Driver service with driver-name = h2 [transactions-repository-1-4z05w] 17:54:18,037 INFO [org.jboss.as.naming] (MSC service thread 1-6) WFLYNAM0003: Starting Naming Service [transactions-repository-1-4z05w] 17:54:18,842 INFO [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-5) WFLYJCA0001: Bound data source [java:jboss/datasources/ExampleDS] [transactions-repository-1-4z05w] 17:54:18,920 WARN [org.infinispan.configuration.cache.IndexingConfigurationBuilder] (ServerService Thread Pool -- 19) ISPN000403: No indexable classes were defined for this indexed cache; switching to autodetection (support for autodetection will be removed in Infinispan 9.0). [transactions-repository-1-4z05w] 17:54:18,934 INFO [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (ServerService Thread Pool -- 19) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be passivated. [transactions-repository-1-4z05w] 17:54:19,094 INFO [org.jboss.as.server.deployment.scanner] (MSC service thread 1-6) WFLYDS0013: Started FileSystemDeploymentService for directory /opt/jboss/infinispan-server/standalone/deployments [transactions-repository-1-4z05w] 17:54:20,553 WARN [org.infinispan.configuration.cache.AbstractStoreConfigurationBuilder] (MSC service thread 1-6) ISPN000149: Fetch persistent state and purge on startup are both disabled, cache may contain stale entries on startup [transactions-repository-1-4z05w] 17:54:20,555 WARN [org.infinispan.configuration.cache.AbstractStoreConfigurationBuilder] (MSC service thread 1-6) ISPN000149: Fetch persistent state and purge on startup are both disabled, cache may contain stale entries on startup [transactions-repository-1-4z05w] [GC (CMS Final Remark) 171687K(1013632K), 0.0585468 secs] [transactions-repository-1-4z05w] 17:54:20,716 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-6) ISPN000078: Starting JGroups channel clustered [transactions-repository-1-4z05w] 17:54:20,732 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-6) ISPN000094: Received new cluster view for channel clustered: [transactions-repository-1-4z05w|2] (3) [transactions-repository-1-4z05w, transactions-repository-1-04x09, transactions-repository-1-1f8dx] [transactions-repository-1-4z05w] 17:54:20,742 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-6) ISPN000079: Channel clustered local address is transactions-repository-1-4z05w, physical addresses are [172.17.0.3:7600] [transactions-repository-1-4z05w] 17:54:20,754 INFO [org.infinispan.globalstate.impl.GlobalStateManagerImpl] (MSC service thread 1-6) ISPN000390: Persisted state, version=9.0.0-SNAPSHOT timestamp=2017-02-18T17:54:20.748Z [transactions-repository-1-4z05w] 17:54:20,755 INFO [org.infinispan.factories.GlobalComponentRegistry] (MSC service thread 1-6) ISPN000128: Infinispan version: Infinispan 'Ruppaner' 9.0.0-SNAPSHOT [transactions-repository-1-4z05w] 17:54:20,933 WARN [org.infinispan.configuration.cache.AbstractStoreConfigurationBuilder] (MSC service thread 1-6) ISPN000149: Fetch persistent state and purge on startup are both disabled, cache may contain stale entries on startup [transactions-repository-1-4z05w] 17:54:20,933 WARN [org.infinispan.configuration.cache.AbstractStoreConfigurationBuilder] (MSC service thread 1-6) ISPN000149: Fetch persistent state and purge on startup are both disabled, cache may contain stale entries on startup [transactions-repository-1-4z05w] 17:54:20,936 WARN [org.infinispan.configuration.cache.IndexingConfigurationBuilder] (MSC service thread 1-5) ISPN000403: No indexable classes were defined for this indexed cache; switching to autodetection (support for autodetection will be removed in Infinispan 9.0). [transactions-repository-1-4z05w] 17:54:20,939 INFO [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (MSC service thread 1-4) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be passivated. [transactions-repository-1-4z05w] 17:54:20,943 INFO [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (MSC service thread 1-4) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be passivated. [transactions-repository-1-4z05w] 17:54:20,954 INFO [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (MSC service thread 1-4) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be passivated. [transactions-repository-1-4z05w] 17:54:20,950 WARN [org.infinispan.configuration.cache.IndexingConfigurationBuilder] (MSC service thread 1-5) ISPN000403: No indexable classes were defined for this indexed cache; switching to autodetection (support for autodetection will be removed in Infinispan 9.0). [transactions-repository-1-4z05w] 17:54:20,956 WARN [org.infinispan.configuration.cache.IndexingConfigurationBuilder] (MSC service thread 1-5) ISPN000403: No indexable classes were defined for this indexed cache; switching to autodetection (support for autodetection will be removed in Infinispan 9.0). [transactions-repository-1-1f8dx] [GC (CMS Final Remark) 172593K(1013632K), 0.0447491 secs] [transactions-repository-1-04x09] 17:54:20,995 WARN [org.infinispan.configuration.cache.AbstractStoreConfigurationBuilder] (MSC service thread 1-3) ISPN000149: Fetch persistent state and purge on startup are both disabled, cache may contain stale entries on startup [transactions-repository-1-04x09] 17:54:20,996 WARN [org.infinispan.configuration.cache.AbstractStoreConfigurationBuilder] (MSC service thread 1-3) ISPN000149: Fetch persistent state and purge on startup are both disabled, cache may contain stale entries on startup [transactions-repository-1-04x09] 17:54:21,109 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-3) ISPN000078: Starting JGroups channel clustered [transactions-repository-1-04x09] 17:54:21,129 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-3) ISPN000094: Received new cluster view for channel clustered: [transactions-repository-1-4z05w|2] (3) [transactions-repository-1-4z05w, transactions-repository-1-04x09, transactions-repository-1-1f8dx] [transactions-repository-1-04x09] 17:54:21,138 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-3) ISPN000079: Channel clustered local address is transactions-repository-1-04x09, physical addresses are [172.17.0.4:7600] [transactions-repository-1-1f8dx] 17:54:21,158 WARN [org.infinispan.configuration.cache.AbstractStoreConfigurationBuilder] (MSC service thread 1-8) ISPN000149: Fetch persistent state and purge on startup are both disabled, cache may contain stale entries on startup [transactions-repository-1-1f8dx] 17:54:21,158 WARN [org.infinispan.configuration.cache.AbstractStoreConfigurationBuilder] (MSC service thread 1-8) ISPN000149: Fetch persistent state and purge on startup are both disabled, cache may contain stale entries on startup [transactions-repository-1-04x09] 17:54:21,160 INFO [org.infinispan.globalstate.impl.GlobalStateManagerImpl] (MSC service thread 1-3) ISPN000390: Persisted state, version=9.0.0-SNAPSHOT timestamp=2017-02-18T17:54:21.156Z [transactions-repository-1-04x09] 17:54:21,160 INFO [org.infinispan.factories.GlobalComponentRegistry] (MSC service thread 1-3) ISPN000128: Infinispan version: Infinispan 'Ruppaner' 9.0.0-SNAPSHOT [transactions-repository-1-1f8dx] 17:54:21,254 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-8) ISPN000078: Starting JGroups channel clustered [transactions-repository-1-1f8dx] 17:54:21,266 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-8) ISPN000094: Received new cluster view for channel clustered: [transactions-repository-1-4z05w|2] (3) [transactions-repository-1-4z05w, transactions-repository-1-04x09, transactions-repository-1-1f8dx] [transactions-repository-1-1f8dx] 17:54:21,269 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-8) ISPN000079: Channel clustered local address is transactions-repository-1-1f8dx, physical addresses are [172.17.0.6:7600] [transactions-repository-1-1f8dx] 17:54:21,282 INFO [org.infinispan.globalstate.impl.GlobalStateManagerImpl] (MSC service thread 1-8) ISPN000390: Persisted state, version=9.0.0-SNAPSHOT timestamp=2017-02-18T17:54:21.279Z [transactions-repository-1-1f8dx] 17:54:21,283 INFO [org.infinispan.factories.GlobalComponentRegistry] (MSC service thread 1-8) ISPN000128: Infinispan version: Infinispan 'Ruppaner' 9.0.0-SNAPSHOT [transactions-repository-1-04x09] 17:54:21,297 WARN [org.infinispan.configuration.cache.AbstractStoreConfigurationBuilder] (MSC service thread 1-3) ISPN000149: Fetch persistent state and purge on startup are both disabled, cache may contain stale entries on startup [transactions-repository-1-04x09] 17:54:21,297 WARN [org.infinispan.configuration.cache.AbstractStoreConfigurationBuilder] (MSC service thread 1-3) ISPN000149: Fetch persistent state and purge on startup are both disabled, cache may contain stale entries on startup [transactions-repository-1-04x09] 17:54:21,300 INFO [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (MSC service thread 1-6) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be passivated. [transactions-repository-1-04x09] 17:54:21,303 WARN [org.infinispan.configuration.cache.IndexingConfigurationBuilder] (MSC service thread 1-2) ISPN000403: No indexable classes were defined for this indexed cache; switching to autodetection (support for autodetection will be removed in Infinispan 9.0). [transactions-repository-1-04x09] 17:54:21,305 INFO [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (MSC service thread 1-6) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be passivated. [transactions-repository-1-04x09] 17:54:21,310 INFO [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (MSC service thread 1-6) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be passivated. [transactions-repository-1-04x09] 17:54:21,310 WARN [org.infinispan.configuration.cache.IndexingConfigurationBuilder] (MSC service thread 1-2) ISPN000403: No indexable classes were defined for this indexed cache; switching to autodetection (support for autodetection will be removed in Infinispan 9.0). [transactions-repository-1-04x09] 17:54:21,311 WARN [org.infinispan.configuration.cache.IndexingConfigurationBuilder] (MSC service thread 1-2) ISPN000403: No indexable classes were defined for this indexed cache; switching to autodetection (support for autodetection will be removed in Infinispan 9.0). [transactions-repository-1-1f8dx] 17:54:21,443 WARN [org.infinispan.configuration.cache.AbstractStoreConfigurationBuilder] (MSC service thread 1-8) ISPN000149: Fetch persistent state and purge on startup are both disabled, cache may contain stale entries on startup [transactions-repository-1-1f8dx] 17:54:21,443 WARN [org.infinispan.configuration.cache.AbstractStoreConfigurationBuilder] (MSC service thread 1-8) ISPN000149: Fetch persistent state and purge on startup are both disabled, cache may contain stale entries on startup [transactions-repository-1-1f8dx] 17:54:21,449 INFO [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (MSC service thread 1-8) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be passivated. [transactions-repository-1-1f8dx] 17:54:21,450 INFO [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (MSC service thread 1-8) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be passivated. [transactions-repository-1-1f8dx] 17:54:21,452 WARN [org.infinispan.configuration.cache.IndexingConfigurationBuilder] (MSC service thread 1-4) ISPN000403: No indexable classes were defined for this indexed cache; switching to autodetection (support for autodetection will be removed in Infinispan 9.0). [transactions-repository-1-1f8dx] 17:54:21,462 WARN [org.infinispan.configuration.cache.IndexingConfigurationBuilder] (MSC service thread 1-4) ISPN000403: No indexable classes were defined for this indexed cache; switching to autodetection (support for autodetection will be removed in Infinispan 9.0). [transactions-repository-1-1f8dx] 17:54:21,462 INFO [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (MSC service thread 1-8) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be passivated. [transactions-repository-1-1f8dx] 17:54:21,468 WARN [org.infinispan.configuration.cache.IndexingConfigurationBuilder] (MSC service thread 1-4) ISPN000403: No indexable classes were defined for this indexed cache; switching to autodetection (support for autodetection will be removed in Infinispan 9.0). [transactions-repository-1-04x09] [GC (CMS Initial Mark) 208341K(1013632K), 0.0101475 secs] [transactions-repository-1-4z05w] 17:54:23,974 ERROR [org.infinispan.topology.ClusterCacheStatus] (transport-thread--p4-t9) ISPN000228: Failed to recover cache transactions state after the current node became the coordinator: java.lang.NullPointerException [transactions-repository-1-4z05w] at org.infinispan.topology.ClusterCacheStatus.updateAvailabilityMode(ClusterCacheStatus.java:147) [transactions-repository-1-4z05w] at org.infinispan.partitionhandling.impl.PreferConsistencyStrategy.updateMembersAndRebalance(PreferConsistencyStrategy.java:259) [transactions-repository-1-4z05w] at org.infinispan.partitionhandling.impl.PreferConsistencyStrategy.onPartitionMerge(PreferConsistencyStrategy.java:210) [transactions-repository-1-4z05w] at org.infinispan.topology.ClusterCacheStatus.doMergePartitions(ClusterCacheStatus.java:554) [transactions-repository-1-4z05w] at org.infinispan.topology.ClusterTopologyManagerImpl.lambda$recoverClusterStatus$3(ClusterTopologyManagerImpl.java:499) [transactions-repository-1-4z05w] at org.infinispan.executors.LimitedExecutor.runTasks(LimitedExecutor.java:144) [transactions-repository-1-4z05w] at org.infinispan.executors.LimitedExecutor.access$100(LimitedExecutor.java:33) [transactions-repository-1-4z05w] at org.infinispan.executors.LimitedExecutor$Runner.run(LimitedExecutor.java:174) [transactions-repository-1-4z05w] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [transactions-repository-1-4z05w] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [transactions-repository-1-4z05w] at java.lang.Thread.run(Thread.java:745) [transactions-repository-1-4z05w] [transactions-repository-1-4z05w] 17:54:23,974 ERROR [org.infinispan.topology.ClusterCacheStatus] (transport-thread--p4-t7) ISPN000228: Failed to recover cache sessions state after the current node became the coordinator: java.lang.NullPointerException [transactions-repository-1-4z05w] at org.infinispan.topology.ClusterCacheStatus.updateAvailabilityMode(ClusterCacheStatus.java:147) [transactions-repository-1-4z05w] at org.infinispan.partitionhandling.impl.PreferConsistencyStrategy.updateMembersAndRebalance(PreferConsistencyStrategy.java:259) [transactions-repository-1-4z05w] at org.infinispan.partitionhandling.impl.PreferConsistencyStrategy.onPartitionMerge(PreferConsistencyStrategy.java:210) [transactions-repository-1-4z05w] at org.infinispan.topology.ClusterCacheStatus.doMergePartitions(ClusterCacheStatus.java:554) [transactions-repository-1-4z05w] at org.infinispan.topology.ClusterTopologyManagerImpl.lambda$recoverClusterStatus$3(ClusterTopologyManagerImpl.java:499) [transactions-repository-1-4z05w] at org.infinispan.executors.LimitedExecutor.runTasks(LimitedExecutor.java:144) [transactions-repository-1-4z05w] at org.infinispan.executors.LimitedExecutor.access$100(LimitedExecutor.java:33) [transactions-repository-1-4z05w] at org.infinispan.executors.LimitedExecutor$Runner.run(LimitedExecutor.java:174) [transactions-repository-1-4z05w] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [transactions-repository-1-4z05w] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [transactions-repository-1-4z05w] at java.lang.Thread.run(Thread.java:745) [transactions-repository-1-4z05w] [transactions-repository-1-4z05w] 17:54:24,005 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-8) DGISPN0001: Started transactions cache from clustered container [transactions-repository-1-4z05w] 17:54:24,005 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-1) DGISPN0001: Started default cache from clustered container [transactions-repository-1-04x09] 17:54:24,005 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-7) DGISPN0001: Started default cache from clustered container [transactions-repository-1-04x09] 17:54:24,006 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-4) DGISPN0001: Started memcachedCache cache from clustered container [transactions-repository-1-04x09] 17:54:24,006 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-1) DGISPN0001: Started transactions cache from clustered container [transactions-repository-1-4z05w] 17:54:24,008 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-7) DGISPN0001: Started memcachedCache cache from clustered container [transactions-repository-1-4z05w] 17:54:24,010 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-6) DGISPN0001: Started sessions cache from clustered container [transactions-repository-1-04x09] 17:54:24,011 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-3) DGISPN0001: Started sessions cache from clustered container [transactions-repository-1-1f8dx] 17:54:24,012 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-5) DGISPN0001: Started sessions cache from clustered container [transactions-repository-1-1f8dx] 17:54:24,012 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-6) DGISPN0001: Started memcachedCache cache from clustered container [transactions-repository-1-1f8dx] 17:54:24,012 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-2) DGISPN0001: Started default cache from clustered container [transactions-repository-1-1f8dx] 17:54:24,013 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-7) DGISPN0001: Started transactions cache from clustered container [transactions-repository-1-4z05w] 17:54:24,036 INFO [org.infinispan.server.endpoint] (MSC service thread 1-7) DGENDPT10000: MemcachedServer starting [transactions-repository-1-4z05w] 17:54:24,039 INFO [org.infinispan.server.endpoint] (MSC service thread 1-3) DGENDPT10000: HotRodServer starting [transactions-repository-1-4z05w] 17:54:24,039 INFO [org.infinispan.server.endpoint] (MSC service thread 1-3) DGENDPT10001: HotRodServer listening on 172.17.0.3:11222 [transactions-repository-1-4z05w] 17:54:24,040 INFO [org.infinispan.server.endpoint] (MSC service thread 1-4) DGENDPT10000: WebSocketServer starting [transactions-repository-1-4z05w] 17:54:24,040 INFO [org.infinispan.server.endpoint] (MSC service thread 1-4) DGENDPT10001: WebSocketServer listening on 172.17.0.3:8181 [transactions-repository-1-1f8dx] 17:54:24,046 INFO [org.infinispan.server.endpoint] (MSC service thread 1-7) DGENDPT10000: WebSocketServer starting [transactions-repository-1-1f8dx] 17:54:24,045 INFO [org.infinispan.server.endpoint] (MSC service thread 1-1) DGENDPT10000: MemcachedServer starting [transactions-repository-1-1f8dx] 17:54:24,046 INFO [org.infinispan.server.endpoint] (MSC service thread 1-4) DGENDPT10000: HotRodServer starting [transactions-repository-1-1f8dx] 17:54:24,047 INFO [org.infinispan.server.endpoint] (MSC service thread 1-4) DGENDPT10001: HotRodServer listening on 172.17.0.6:11222 [transactions-repository-1-1f8dx] 17:54:24,048 INFO [org.infinispan.server.endpoint] (MSC service thread 1-1) DGENDPT10001: MemcachedServer listening on 172.17.0.6:11211 [transactions-repository-1-04x09] 17:54:24,051 INFO [org.infinispan.server.endpoint] (MSC service thread 1-3) DGENDPT10000: MemcachedServer starting [transactions-repository-1-04x09] 17:54:24,051 INFO [org.infinispan.server.endpoint] (MSC service thread 1-4) DGENDPT10000: WebSocketServer starting [transactions-repository-1-04x09] 17:54:24,052 INFO [org.infinispan.server.endpoint] (MSC service thread 1-1) DGENDPT10000: HotRodServer starting [transactions-repository-1-04x09] 17:54:24,055 INFO [org.infinispan.server.endpoint] (MSC service thread 1-1) DGENDPT10001: HotRodServer listening on 172.17.0.4:11222 [transactions-repository-1-04x09] 17:54:24,055 INFO [org.infinispan.server.endpoint] (MSC service thread 1-3) DGENDPT10001: MemcachedServer listening on 172.17.0.4:11211 [transactions-repository-1-04x09] 17:54:24,056 INFO [org.infinispan.server.endpoint] (MSC service thread 1-4) DGENDPT10001: WebSocketServer listening on 172.17.0.4:8181 [transactions-repository-1-4z05w] 17:54:24,063 INFO [org.infinispan.server.endpoint] (MSC service thread 1-7) DGENDPT10001: MemcachedServer listening on 172.17.0.3:11211 [transactions-repository-1-1f8dx] 17:54:24,059 INFO [org.infinispan.server.endpoint] (MSC service thread 1-7) DGENDPT10001: WebSocketServer listening on 172.17.0.6:8181 [transactions-repository-1-4z05w] 17:54:24,070 INFO [org.infinispan.server.endpoint] (MSC service thread 1-1) DGENDPT10000: REST starting [transactions-repository-1-1f8dx] 17:54:24,079 INFO [org.infinispan.server.endpoint] (MSC service thread 1-6) DGENDPT10000: REST starting [transactions-repository-1-04x09] 17:54:24,071 INFO [org.infinispan.server.endpoint] (MSC service thread 1-8) DGENDPT10000: REST starting [transactions-repository-1-4z05w] 17:54:24,193 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t20) ISPN000310: Starting cluster-wide rebalance for cache ___protobuf_metadata, topology CacheTopology{id=2, rebalanceId=2, currentCH=ReplicatedConsistentHash{ns = 256, owners = (1)[transactions-repository-1-4z05w: 256]}, pendingCH=ReplicatedConsistentHash{ns = 256, owners = (2)[transactions-repository-1-4z05w: 126, transactions-repository-1-1f8dx: 130]}, unionCH=null, actualMembers=[transactions-repository-1-4z05w, transactions-repository-1-1f8dx], persistentUUIDs=[45bed168-6b22-4cc9-b164-6a8b3fe6367f, ced31139-f1a1-4ee8-96af-f37c5386ecc0]} [transactions-repository-1-4z05w] 17:54:24,197 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t20) [Context=___protobuf_metadata][Context=transactions-repository-1-4z05w]ISPN100002: Started local rebalance [transactions-repository-1-4z05w] 17:54:24,246 INFO [org.infinispan.CLUSTER] (transport-thread--p4-t24) [Context=___protobuf_metadata][Context=transactions-repository-1-4z05w]ISPN100003: Finished local rebalance [transactions-repository-1-4z05w] 17:54:24,255 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t22) ISPN000310: Starting cluster-wide rebalance for cache ___script_cache, topology CacheTopology{id=2, rebalanceId=2, currentCH=ReplicatedConsistentHash{ns = 256, owners = (1)[transactions-repository-1-4z05w: 256]}, pendingCH=ReplicatedConsistentHash{ns = 256, owners = (2)[transactions-repository-1-4z05w: 126, transactions-repository-1-1f8dx: 130]}, unionCH=null, actualMembers=[transactions-repository-1-4z05w, transactions-repository-1-1f8dx], persistentUUIDs=[45bed168-6b22-4cc9-b164-6a8b3fe6367f, ced31139-f1a1-4ee8-96af-f37c5386ecc0]} [transactions-repository-1-4z05w] 17:54:24,256 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t22) [Context=___script_cache][Context=transactions-repository-1-4z05w]ISPN100002: Started local rebalance [transactions-repository-1-4z05w] 17:54:24,272 INFO [org.infinispan.CLUSTER] (transport-thread--p4-t2) [Context=___script_cache][Context=transactions-repository-1-4z05w]ISPN100003: Finished local rebalance [transactions-repository-1-4z05w] 17:54:24,317 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t2) ISPN000310: Starting cluster-wide rebalance for cache ___hotRodTopologyCache, topology CacheTopology{id=2, rebalanceId=2, currentCH=ReplicatedConsistentHash{ns = 256, owners = (1)[transactions-repository-1-4z05w: 256]}, pendingCH=ReplicatedConsistentHash{ns = 256, owners = (2)[transactions-repository-1-4z05w: 134, transactions-repository-1-04x09: 122]}, unionCH=null, actualMembers=[transactions-repository-1-4z05w, transactions-repository-1-04x09], persistentUUIDs=[45bed168-6b22-4cc9-b164-6a8b3fe6367f, 5d246b27-021c-4425-b3c4-14017f378635]} [transactions-repository-1-4z05w] 17:54:24,318 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t2) [Context=___hotRodTopologyCache][Context=transactions-repository-1-4z05w]ISPN100002: Started local rebalance [transactions-repository-1-4z05w] 17:54:24,321 INFO [org.infinispan.CLUSTER] (transport-thread--p4-t11) [Context=___hotRodTopologyCache][Context=transactions-repository-1-4z05w]ISPN100003: Finished local rebalance [transactions-repository-1-4z05w] 17:54:24,364 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t8) [Context=___protobuf_metadata][Context=transactions-repository-1-1f8dx]ISPN100003: Finished local rebalance [transactions-repository-1-4z05w] 17:54:24,366 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t10) [Context=___script_cache][Context=transactions-repository-1-1f8dx]ISPN100003: Finished local rebalance [transactions-repository-1-4z05w] 17:54:24,382 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t6) [Context=___hotRodTopologyCache][Context=transactions-repository-1-04x09]ISPN100003: Finished local rebalance [transactions-repository-1-4z05w] 17:54:24,385 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t8) ISPN000336: Finished cluster-wide rebalance for cache ___protobuf_metadata, topology id = 2 [transactions-repository-1-4z05w] 17:54:24,387 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t8) ISPN000310: Starting cluster-wide rebalance for cache ___protobuf_metadata, topology CacheTopology{id=4, rebalanceId=3, currentCH=ReplicatedConsistentHash{ns = 256, owners = (2)[transactions-repository-1-4z05w: 126, transactions-repository-1-1f8dx: 130]}, pendingCH=ReplicatedConsistentHash{ns = 256, owners = (3)[transactions-repository-1-4z05w: 90, transactions-repository-1-1f8dx: 86, transactions-repository-1-04x09: 80]}, unionCH=null, actualMembers=[transactions-repository-1-4z05w, transactions-repository-1-1f8dx, transactions-repository-1-04x09], persistentUUIDs=[45bed168-6b22-4cc9-b164-6a8b3fe6367f, ced31139-f1a1-4ee8-96af-f37c5386ecc0, 5d246b27-021c-4425-b3c4-14017f378635]} [transactions-repository-1-4z05w] 17:54:24,388 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t8) [Context=___protobuf_metadata][Context=transactions-repository-1-4z05w]ISPN100002: Started local rebalance [transactions-repository-1-4z05w] 17:54:24,394 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t6) ISPN000336: Finished cluster-wide rebalance for cache ___hotRodTopologyCache, topology id = 2 [transactions-repository-1-4z05w] 17:54:24,395 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t10) ISPN000336: Finished cluster-wide rebalance for cache ___script_cache, topology id = 2 [transactions-repository-1-4z05w] 17:54:24,397 INFO [org.infinispan.CLUSTER] (transport-thread--p4-t19) [Context=___protobuf_metadata][Context=transactions-repository-1-4z05w]ISPN100003: Finished local rebalance [transactions-repository-1-4z05w] 17:54:24,397 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t7) [Context=___protobuf_metadata][Context=transactions-repository-1-1f8dx]ISPN100003: Finished local rebalance [transactions-repository-1-4z05w] 17:54:24,398 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t10) ISPN000310: Starting cluster-wide rebalance for cache ___script_cache, topology CacheTopology{id=4, rebalanceId=3, currentCH=ReplicatedConsistentHash{ns = 256, owners = (2)[transactions-repository-1-4z05w: 126, transactions-repository-1-1f8dx: 130]}, pendingCH=ReplicatedConsistentHash{ns = 256, owners = (3)[transactions-repository-1-4z05w: 90, transactions-repository-1-1f8dx: 86, transactions-repository-1-04x09: 80]}, unionCH=null, actualMembers=[transactions-repository-1-4z05w, transactions-repository-1-1f8dx, transactions-repository-1-04x09], persistentUUIDs=[45bed168-6b22-4cc9-b164-6a8b3fe6367f, ced31139-f1a1-4ee8-96af-f37c5386ecc0, 5d246b27-021c-4425-b3c4-14017f378635]} [transactions-repository-1-4z05w] 17:54:24,398 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t10) [Context=___script_cache][Context=transactions-repository-1-4z05w]ISPN100002: Started local rebalance [transactions-repository-1-4z05w] 17:54:24,401 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t6) ISPN000310: Starting cluster-wide rebalance for cache ___hotRodTopologyCache, topology CacheTopology{id=4, rebalanceId=3, currentCH=ReplicatedConsistentHash{ns = 256, owners = (2)[transactions-repository-1-4z05w: 134, transactions-repository-1-04x09: 122]}, pendingCH=ReplicatedConsistentHash{ns = 256, owners = (3)[transactions-repository-1-4z05w: 90, transactions-repository-1-04x09: 80, transactions-repository-1-1f8dx: 86]}, unionCH=null, actualMembers=[transactions-repository-1-4z05w, transactions-repository-1-04x09, transactions-repository-1-1f8dx], persistentUUIDs=[45bed168-6b22-4cc9-b164-6a8b3fe6367f, 5d246b27-021c-4425-b3c4-14017f378635, ced31139-f1a1-4ee8-96af-f37c5386ecc0]} [transactions-repository-1-4z05w] 17:54:24,405 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t6) [Context=___hotRodTopologyCache][Context=transactions-repository-1-4z05w]ISPN100002: Started local rebalance [transactions-repository-1-4z05w] 17:54:24,407 INFO [org.infinispan.CLUSTER] (transport-thread--p4-t20) [Context=___hotRodTopologyCache][Context=transactions-repository-1-4z05w]ISPN100003: Finished local rebalance [transactions-repository-1-4z05w] 17:54:24,409 INFO [org.infinispan.CLUSTER] (transport-thread--p4-t3) [Context=___script_cache][Context=transactions-repository-1-4z05w]ISPN100003: Finished local rebalance [transactions-repository-1-4z05w] 17:54:24,420 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t11) [Context=___script_cache][Context=transactions-repository-1-1f8dx]ISPN100003: Finished local rebalance [transactions-repository-1-4z05w] 17:54:24,425 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t11) [Context=___hotRodTopologyCache][Context=transactions-repository-1-04x09]ISPN100003: Finished local rebalance [transactions-repository-1-4z05w] 17:54:24,476 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t17) [Context=___hotRodTopologyCache][Context=transactions-repository-1-1f8dx]ISPN100003: Finished local rebalance [transactions-repository-1-4z05w] 17:54:24,480 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t17) ISPN000336: Finished cluster-wide rebalance for cache ___hotRodTopologyCache, topology id = 4 [transactions-repository-1-4z05w] 17:54:24,536 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t21) [Context=___protobuf_metadata][Context=transactions-repository-1-04x09]ISPN100003: Finished local rebalance [transactions-repository-1-4z05w] 17:54:24,537 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t21) ISPN000336: Finished cluster-wide rebalance for cache ___protobuf_metadata, topology id = 4 [transactions-repository-1-4z05w] 17:54:24,544 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t24) [Context=___script_cache][Context=transactions-repository-1-04x09]ISPN100003: Finished local rebalance [transactions-repository-1-4z05w] 17:54:24,546 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t24) ISPN000336: Finished cluster-wide rebalance for cache ___script_cache, topology id = 4 [transactions-repository-1-4z05w] 17:54:24,567 INFO [org.infinispan.rest.NettyRestServer] (MSC service thread 1-1) ISPN012003: REST server starting, listening on 172.17.0.3:8080 [transactions-repository-1-4z05w] 17:54:24,568 INFO [org.infinispan.server.endpoint] (MSC service thread 1-1) DGENDPT10002: REST mapped to /rest [transactions-repository-1-1f8dx] 17:54:24,576 INFO [org.infinispan.rest.NettyRestServer] (MSC service thread 1-6) ISPN012003: REST server starting, listening on 172.17.0.6:8080 [transactions-repository-1-1f8dx] 17:54:24,577 INFO [org.infinispan.server.endpoint] (MSC service thread 1-6) DGENDPT10002: REST mapped to /rest [transactions-repository-1-04x09] 17:54:24,639 INFO [org.infinispan.rest.NettyRestServer] (MSC service thread 1-8) ISPN012003: REST server starting, listening on 172.17.0.4:8080 [transactions-repository-1-04x09] 17:54:24,639 INFO [org.infinispan.server.endpoint] (MSC service thread 1-8) DGENDPT10002: REST mapped to /rest [transactions-repository-1-1f8dx] [GC (Allocation Failure) 309342K->63160K(1013632K), 0.1823599 secs] [transactions-repository-1-4z05w] [GC (Allocation Failure) 309352K->65174K(1013632K), 0.1616480 secs] [transactions-repository-1-04x09] [GC (Allocation Failure) 309268K->61649K(1013632K), 0.1507677 secs] [transactions-repository-1-1f8dx] 17:54:25,021 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0060: Http management interface listening on http://0.0.0.0:9990/management [transactions-repository-1-1f8dx] 17:54:25,022 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0051: Admin console listening on http://0.0.0.0:9990 [transactions-repository-1-1f8dx] 17:54:25,022 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0025: Infinispan Server 9.0.0-SNAPSHOT (WildFly Core 2.2.0.Final) started in 9718ms - Started 197 of 282 services (164 services are lazy, passive or on-demand) [transactions-repository-1-4z05w] 17:54:25,021 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0060: Http management interface listening on http://0.0.0.0:9990/management [transactions-repository-1-4z05w] 17:54:25,024 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0051: Admin console listening on http://0.0.0.0:9990 [transactions-repository-1-4z05w] 17:54:25,027 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0025: Infinispan Server 9.0.0-SNAPSHOT (WildFly Core 2.2.0.Final) started in 9643ms - Started 197 of 282 services (164 services are lazy, passive or on-demand) [transactions-repository-1-04x09] 17:54:25,063 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0060: Http management interface listening on http://0.0.0.0:9990/management [transactions-repository-1-04x09] 17:54:25,064 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0051: Admin console listening on http://0.0.0.0:9990 [transactions-repository-1-04x09] 17:54:25,064 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0025: Infinispan Server 9.0.0-SNAPSHOT (WildFly Core 2.2.0.Final) started in 9746ms - Started 197 of 282 services (164 services are lazy, passive or on-demand) [transactions-repository-1-04x09] [GC (CMS Final Remark) 89552K(1013632K), 0.0310821 secs] {noformat} Note that I applied multiplexed logs from 3 nodes. I hope this will be helpful for diagnosing. Steps to reproduce: * Start local OpenShift Cluster * invoke `./init_infrastructure.sh` from https://github.com/slaskawi/presentations/tree/ISPN-7487-reproducer * Kill some nodes * Start them again -- This message was sent by Atlassian JIRA (v7.2.3#72005)